Sep 15 07:16:22.351034 ip-10-0-141-227 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 07:16:22.351045 ip-10-0-141-227 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 07:16:22.351052 ip-10-0-141-227 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 07:16:22.351196 ip-10-0-141-227 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 07:16:33.720193 ip-10-0-141-227 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 07:16:33.720208 ip-10-0-141-227 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e2dc28b250d449d98a3a1bf7051d3463 --
Sep 15 07:18:31.805375 ip-10-0-141-227 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 07:18:32.152412 ip-10-0-141-227 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:32.152412 ip-10-0-141-227 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 07:18:32.152412 ip-10-0-141-227 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:32.152412 ip-10-0-141-227 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 07:18:32.152412 ip-10-0-141-227 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 07:18:32.153423 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.153357    2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 07:18:32.157296 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157282    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.157296 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157296    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157299    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157303    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157306    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157309    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157312    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157315    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157317    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157320    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157323    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157326    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157335    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157338    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157341    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157344    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157347    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157350    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157352    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157355    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157357    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.157361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157360    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157363    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157365    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157368    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157373    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157377    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157379    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157382    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157385    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157387    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157390    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157393    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157395    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157398    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157402    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157406    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157408    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157411    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157414    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.157831 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157416    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157418    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157421    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157423    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157426    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157429    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157432    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157434    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157437    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157440    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157442    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157445    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157448    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157450    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157453    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157456    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157458    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157460    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157463    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157466    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.158323 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157468    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157470    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157473    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157475    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157478    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157480    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157483    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157485    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157488    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157491    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157493    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157495    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157498    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157503    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157506    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157509    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157511    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157514    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157521    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.158805 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157524    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157527    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157530    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157533    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157535    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157538    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157541    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157544    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157547    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157550    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157553    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157556    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157558    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157561    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157564    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157566    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157569    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157572    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157575    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.157577    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.159268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158244    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158250    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158253    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158255    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158258    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158261    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158264    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158266    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158269    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158271    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158274    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158277    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158281    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158284    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158286    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158289    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158291    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158294    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158298    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.159740 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158301    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158304    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158307    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158309    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158312    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158314    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158317    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158320    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158322    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158325    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158328    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158331    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158333    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158336    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158339    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158341    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158343    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158346    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158348    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158351    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.160234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158354    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158356    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158360    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158364    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158368    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158371    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158374    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158377    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158379    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158382    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158384    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158387    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158389    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158392    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158395    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158397    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158399    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158402    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158404    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.160741 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158407    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158409    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158413    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158415    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158418    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158420    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158423    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158425    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158428    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158430    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158433    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158436    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158439    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158441    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158444    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158447    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158449    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158452    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158455    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158457    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.161233 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158460    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158463    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158467    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158470    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158472    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158475    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158477    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158480    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158482    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158484    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158487    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158489    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158492    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158494    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158497    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158499    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158502    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158504    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158507    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.161749 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158509    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158513    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158852    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158857    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158860    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158863    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158866    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158869    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158871    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158874    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158877    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158880    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158883    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158886    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158888    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158891    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158893    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158896    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158898    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.162250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158901    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158903    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158906    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158908    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158911    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158913    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158916    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158918    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158921    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158923    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158926    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158928    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158931    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158933    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158936    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158938    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158941    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158944    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158946    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158949    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.162718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158952    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158954    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158958    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158962    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158965    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158968    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158971    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158974    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158978    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158981    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158984    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158987    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158990    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158992    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158995    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.158998    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159001    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159003    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159006    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.163250 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159008    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159012    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159015    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159019    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159021    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159024    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159026    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159029    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159031    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159034    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159036    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159039    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159041    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159044    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159046    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159049    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159052    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159054    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159057    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.163713 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159060    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159062    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159066    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159068    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159071    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159073    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159076    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159079    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159081    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159084    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159086    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159089    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159091    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159093    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159096    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159099    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159101    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159104    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159107    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159109    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.164199 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159125    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159129    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159131    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159134    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159197    2574 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159236    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159242    2574 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159246    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159249    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159253    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159256    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159260    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159263    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159266    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159269    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159274    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159277    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159280    2574 flags.go:64] FLAG: --cgroup-root=""
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159283    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159286    2574 flags.go:64] FLAG: --client-ca-file=""
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159289    2574 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159292    2574 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159296    2574 flags.go:64] FLAG: --cluster-domain=""
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159299    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 07:18:32.164946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159302    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159305    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159309    2574 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159312    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159315    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159318    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159322    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159325    2574 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159328    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159331    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159334    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159337    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159341    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159344    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159346    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159349    2574 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159352    2574 flags.go:64] FLAG: --enable-server="true"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159355    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159359    2574 flags.go:64] FLAG: --event-burst="100"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159362    2574 flags.go:64] FLAG: --event-qps="50"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159365    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159368    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159371    2574 flags.go:64] FLAG: --eviction-hard=""
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159374    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 07:18:32.165924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159383    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159387    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159390    2574 flags.go:64] FLAG: --eviction-soft=""
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159393    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159395    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159398    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159401    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159404    2574 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159407    2574 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159410    2574 flags.go:64] FLAG: --feature-gates=""
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159418    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159421    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159426    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159429    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159432    2574 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159435    2574 flags.go:64] FLAG: --help="false"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159438    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159441    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159444    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159447    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159450    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159453    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159456    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 07:18:32.166906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159459    2574 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159462    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159464    2574 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159467    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159471    2574 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159473    2574 flags.go:64] FLAG: --kube-reserved=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159476    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159479    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159482    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159486    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159490    2574 flags.go:64] FLAG: --lock-file=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159492    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159495    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159498    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159503    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159506    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159509    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159512    2574 flags.go:64] FLAG: --logging-format="text"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159514    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159518    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159521    2574 flags.go:64] FLAG: --manifest-url=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159525    2574 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159529    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159532    2574 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159536    2574 flags.go:64] FLAG: --max-pods="110"
Sep 15 07:18:32.167651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159539    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159542    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159545    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159548    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159551    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159554    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159557    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159563    2574 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159566    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159569    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159572    2574 flags.go:64] FLAG: --pod-cidr=""
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159575    2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159580    2574 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159583    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159586    2574 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159589    2574 flags.go:64] FLAG: --port="10250"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159593    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159596    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0851e23c139825392"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159599    2574 flags.go:64] FLAG: --qos-reserved=""
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159602    2574 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159605    2574 flags.go:64] FLAG: --register-node="true"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159608    2574 flags.go:64] FLAG: --register-with-taints=""
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159611    2574 flags.go:64] FLAG: --registry-burst="10"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159614    2574 flags.go:64] FLAG: --registry-qps="5"
Sep 15 07:18:32.168312 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159617    2574 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159620    2574 flags.go:64] FLAG: --reserved-memory=""
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159623    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159626    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159629    2574 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159633    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159636    2574 flags.go:64] FLAG: --runonce="false"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159639    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159642    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159645    2574 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159648    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159651    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159654    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159657    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159659    2574 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159662    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159665    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159668    2574 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159671    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159674    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159677    2574 flags.go:64] FLAG: --system-cgroups=""
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159680    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159685    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159687    2574 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159692    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 07:18:32.168915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159696    2574 flags.go:64] FLAG: --tls-min-version=""
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159699    2574 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159702    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159705    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159707    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159710    2574 flags.go:64] FLAG: --v="2"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159714    2574 flags.go:64] FLAG: --version="false"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159717    2574 flags.go:64] FLAG: --vmodule=""
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159720    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.159724    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159825    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159829    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159835    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159838    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159841    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159843    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159847    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159850    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159854    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159857    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159860    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159863    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.169580 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159866    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159869    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159872    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159874    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159877    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159879    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159882    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159884    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159887    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159890    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159893    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159896    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159898    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159901    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159904    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159906    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159909    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159911    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159914    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159916    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.170129 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159919    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159921    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159925    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159927    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159930    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159932    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159935    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159937    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159940    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159942    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159945    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159948    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159950    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159953    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159955    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159958    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159961    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159963    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159966    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159968    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.170622 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159971    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159974    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159976    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159980    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159982    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159984    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159988    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159990    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159993    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159995    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.159998    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160001    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160005    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160008    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160012    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160014    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160017    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160019    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160023    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.171100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160027    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160030    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160033    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160036    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160038    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160041    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160044    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160047    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160049    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160052    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160055    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160057    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160060    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160063    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160065    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160068    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160071    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160073    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160076    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.171635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160078    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160081    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160084    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160086    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160089    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160091    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160094    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160096    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.160100    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.160639    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.168713    2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.168729    2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.168747    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.168752    2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168804    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.172105 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168809    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168812    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168816    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168819    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168823    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168827    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168830    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168834    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168837    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168840    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168842    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168845    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168848    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168850    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168853    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168855    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168858    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168863    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.172533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168866    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168869    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168872    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168875    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168879    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168882    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168885    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168888    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168891    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168893    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168897    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168900    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168903    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168906    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168909    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168912    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168915    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168918    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168921    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.172979 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168923    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168926    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168930    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168933    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168936    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168939    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168942    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168945    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168947    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168950    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168952    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168955    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168958    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168960    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168963    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168966    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168969    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168972    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168974    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168977    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.173458 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168979    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168982    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168984    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168988    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168991    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168994    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168996    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.168999    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169002    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169004    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169007    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169009    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169012    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169015    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169018    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169020    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169023    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169025    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169028    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169030    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.173938 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169033    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169036    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169038    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169041    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169043    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169046    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169048    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169051    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169053    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169056    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169058    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169061    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169063    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169067    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169069    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169072    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169075    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169077    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169080    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169083    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.174438 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169086    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.169091    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169219    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169225    2574 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169229    2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169233    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169236    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169239    2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169242    2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169244    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169247    2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169250    2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169252    2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169255    2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 07:18:32.174916 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169259    2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169262    2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169265    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169267    2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169270    2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169273    2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169276    2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169278    2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169281    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169285    2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169289    2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169291    2574 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169295    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169298    2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169301    2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169303    2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169306    2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169309    2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169311    2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169314    2574 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 07:18:32.175282 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169316    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169319    2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169321    2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169324    2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169326    2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169329    2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169332    2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169334    2574 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169337    2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169339    2574 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169342    2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169344    2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169347    2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169350    2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169352    2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169355    2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169359    2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169362    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169365    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 07:18:32.175766 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169368    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169371    2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169373    2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169376    2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169379    2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169382    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169385    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169388    2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169391    2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169394    2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169397    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169399    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169403    2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169405    2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169408    2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169411    2574 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169413    2574 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169416    2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169419    2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169421    2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 07:18:32.176235 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169424    2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169426    2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169429    2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169432    2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169434    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169437    2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169439    2574 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169443    2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169445    2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169448    2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169451    2574 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169453    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169456    2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169459    2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169461    2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169464    2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169466    2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169469    2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169472    2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169474    2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 07:18:32.176733 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169477    2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169480    2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169482    2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169485    2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169488    2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169491    2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169493    2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:32.169496    2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.169500    2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.169596    2574 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.171429    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.172182    2574 server.go:1019] "Starting client certificate rotation"
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.172279    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 07:18:32.177294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.172884    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 07:18:32.177619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.174627    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 07:18:32.177619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.177135    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 07:18:32.186243 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.186225    2574 log.go:25] "Validated CRI v1 runtime API"
Sep 15 07:18:32.189024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.189007    2574 log.go:25] "Validated CRI v1 image API"
Sep 15 07:18:32.189837 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.189819    2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 07:18:32.195348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.195327    2574 fs.go:135] Filesystem UUIDs: map[338631d5-3223-45a8-b844-5bf8c63bcc87:/dev/nvme0n1p3 343afecf-d628-4ce0-bcab-a002626f462c:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 15 07:18:32.195423 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.195346    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 07:18:32.201111 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.201005    2574 manager.go:216] Machine: {Timestamp:2026-09-15 07:18:32.199876967 +0000 UTC m=+0.294043221 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101954 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b65654a51b647b7ed7e21ccb442cc SystemUUID:ec2b6565-4a51-b647-b7ed-7e21ccb442cc BootID:e2dc28b2-50d4-49d9-8a3a-1bf7051d3463 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f9:97:0c:a9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f9:97:0c:a9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:42:f0:ca:f3:2e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 07:18:32.201111 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.201104    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 07:18:32.201223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.201207    2574 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 07:18:32.203952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.203928    2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 07:18:32.204082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.203953    2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-227.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 07:18:32.204540 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.204522    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 07:18:32.205069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.205059    2574 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 07:18:32.205101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.205072    2574 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 07:18:32.205101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.205080    2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 07:18:32.205699 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.205689    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 07:18:32.206432 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.206422    2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 07:18:32.209139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209128    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 07:18:32.209246 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209237    2574 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 07:18:32.209284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209264    2574 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 07:18:32.209284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209273    2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 07:18:32.209348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209287    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 07:18:32.209348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209295    2574 kubelet.go:398] "Adding apiserver pod source"
Sep 15 07:18:32.209348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209303    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 07:18:32.209617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209604    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 07:18:32.209689 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.209625    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 07:18:32.212519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.212502    2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 07:18:32.215015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.215001    2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 07:18:32.215074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.215030    2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 07:18:32.217106 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217093    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217109    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217128    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217134    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217145    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217152    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217164    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217169    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 07:18:32.217186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217186    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 07:18:32.217534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217194    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 07:18:32.217534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217204    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 07:18:32.217534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217222    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 07:18:32.217889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217878    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 07:18:32.217889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.217888    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 07:18:32.219607 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.219588    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 07:18:32.219696 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.219649    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 07:18:32.219734 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.219723    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-227.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 07:18:32.221361 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.221346    2574 server.go:1268] "Started kubelet"
Sep 15 07:18:32.221448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.221424    2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 07:18:32.221920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.221485    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 07:18:32.221920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.221619    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 07:18:32.222200 ip-10-0-141-227 systemd[1]: Started Kubernetes Kubelet.
Sep 15 07:18:32.222549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.222535    2574 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 07:18:32.222619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.222563    2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 07:18:32.228047 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.227984    2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 07:18:32.231461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.231445    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 07:18:32.231937 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.231919    2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 07:18:32.232073 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.232057    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.232486 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232469    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 07:18:32.232594 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232471    2574 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 07:18:32.232594 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232594    2574 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 07:18:32.232736 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232701    2574 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 07:18:32.232736 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232712    2574 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 07:18:32.232967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232954    2574 factory.go:153] Registering CRI-O factory
Sep 15 07:18:32.233010 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.232994    2574 factory.go:223] Registration of the crio container factory successfully
Sep 15 07:18:32.233055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233038    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 07:18:32.233055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233044    2574 factory.go:55] Registering systemd factory
Sep 15 07:18:32.233055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233049    2574 factory.go:223] Registration of the systemd container factory successfully
Sep 15 07:18:32.233183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233060    2574 factory.go:103] Registering Raw factory
Sep 15 07:18:32.233183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233069    2574 manager.go:1196] Started watching for new ooms in manager
Sep 15 07:18:32.233450 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.233438    2574 manager.go:349] Starting recovery of all containers
Sep 15 07:18:32.235095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.235071    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvmbp"
Sep 15 07:18:32.237839 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.237815    2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 07:18:32.237935 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.237886    2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-227.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 07:18:32.238291 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.237256    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-227.ec2.internal.18d56d6618e3e600  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-227.ec2.internal,UID:ip-10-0-141-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-227.ec2.internal,},FirstTimestamp:2026-09-15 07:18:32.221320704 +0000 UTC m=+0.315486957,LastTimestamp:2026-09-15 07:18:32.221320704 +0000 UTC m=+0.315486957,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-227.ec2.internal,}"
Sep 15 07:18:32.243942 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.243926    2574 manager.go:354] Recovery completed
Sep 15 07:18:32.246179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.246162    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pvmbp"
Sep 15 07:18:32.250206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.250193    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.252253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252236    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.252331 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252264    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.252331 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252273    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.252830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252815    2574 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 07:18:32.252830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252829    2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 07:18:32.252934 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252845    2574 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 07:18:32.252934 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.252860    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.253567 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253552    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.253629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253580    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.253629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253593    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.253967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253956    2574 policy_none.go:49] "None policy: Start"
Sep 15 07:18:32.254014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253972    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 07:18:32.254014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.253984    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 07:18:32.254343 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.254260    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-227.ec2.internal.18d56d661abbe04b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-227.ec2.internal,UID:ip-10-0-141-227.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-227.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-227.ec2.internal,},FirstTimestamp:2026-09-15 07:18:32.252252235 +0000 UTC m=+0.346418488,LastTimestamp:2026-09-15 07:18:32.252252235 +0000 UTC m=+0.346418488,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-227.ec2.internal,}"
Sep 15 07:18:32.255534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.255522    2574 policy_none.go:47] "Start"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.294863    2574 manager.go:337] "Starting Device Plugin manager"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.294986    2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.294997    2574 server.go:87] "Starting device plugin registration server"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.295226    2574 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.295238    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.295362    2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.295434    2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.295447    2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.296103    2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 07:18:32.299487 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.296180    2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.370558 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.370530    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 07:18:32.371755 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.371738    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 07:18:32.371831 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.371779    2574 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 07:18:32.371831 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.371799    2574 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 07:18:32.371906 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.371863    2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 07:18:32.374568 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.374551    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:32.395562 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.395547    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.396422 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.396402    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.396523 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.396437    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.396523 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.396449    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.396523 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.396468    2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.403702 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.403647    2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.403702 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.403665    2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-227.ec2.internal\": node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.419940 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.419926    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.472851 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.472824    2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"]
Sep 15 07:18:32.472966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.472891    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.474217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.474203    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.474281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.474228    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.474281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.474237    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.475570 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.475559    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.475709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.475695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.475746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.475725    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.476207 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476192    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.476285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476216    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.476285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476230    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.476391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476296    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.476391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476319    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.476391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.476333    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.477555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.477539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.477631 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.477569    2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 07:18:32.478245 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.478230    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 07:18:32.478342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.478255    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 07:18:32.478342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.478268    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeHasSufficientPID"
Sep 15 07:18:32.491662 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.491648    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-227.ec2.internal\" not found" node="ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.495561 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.495547    2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-227.ec2.internal\" not found" node="ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.520564 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.520548    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.534422 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.534406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.534483 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.534428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.534483 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.534445    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9f17e721a455587375069a20c2a5ed0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-227.ec2.internal\" (UID: \"d9f17e721a455587375069a20c2a5ed0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.620630 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.620614    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.634951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.634931    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.635034 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.634961    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9f17e721a455587375069a20c2a5ed0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-227.ec2.internal\" (UID: \"d9f17e721a455587375069a20c2a5ed0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.635034 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.634987    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.635034 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.635022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.635171 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.635031    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/193dba6a0f198e81f9ade158f96f4c19-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal\" (UID: \"193dba6a0f198e81f9ade158f96f4c19\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.635171 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.635036    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d9f17e721a455587375069a20c2a5ed0-config\") pod \"kube-apiserver-proxy-ip-10-0-141-227.ec2.internal\" (UID: \"d9f17e721a455587375069a20c2a5ed0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.721355 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.721309    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.794854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.794828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.797772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:32.797756    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:32.821618 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.821598    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:32.922031 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:32.922010    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.022568 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.022514    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.123036 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.123014    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.173069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.173051    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 07:18:33.173705 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.173254    2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 07:18:33.223859 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.223833    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.249927 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.249892    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 07:13:32 +0000 UTC" deadline="2028-02-28 13:32:22.804797274 +0000 UTC"
Sep 15 07:18:33.249927 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.249922    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12750h13m49.554878724s"
Sep 15 07:18:33.268270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.268245    2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:33.324784 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.324730    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.426421 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.426386    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.481814 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:33.481783    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod193dba6a0f198e81f9ade158f96f4c19.slice/crio-1e0c6ff669ab4339513d10cfdb395d1f78b6325b9b6642dfcfc75db96593a217 WatchSource:0}: Error finding container 1e0c6ff669ab4339513d10cfdb395d1f78b6325b9b6642dfcfc75db96593a217: Status 404 returned error can't find the container with id 1e0c6ff669ab4339513d10cfdb395d1f78b6325b9b6642dfcfc75db96593a217
Sep 15 07:18:33.485264 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.485248    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:18:33.527091 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.527060    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.581242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.581194    2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:33.628767 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:33.628741    2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-227.ec2.internal\" not found"
Sep 15 07:18:33.688299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.688273    2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:33.733233 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.733210    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:33.743799 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.743778    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 07:18:33.744774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.744759    2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal"
Sep 15 07:18:33.750621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:33.750607    2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 07:18:34.054942 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.054881    2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 07:18:34.127132 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.127072    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f17e721a455587375069a20c2a5ed0.slice/crio-fb1ea6457eec340119ee1b32960da5e730b7e2ecad9ed52812ccae991915050a WatchSource:0}: Error finding container fb1ea6457eec340119ee1b32960da5e730b7e2ecad9ed52812ccae991915050a: Status 404 returned error can't find the container with id fb1ea6457eec340119ee1b32960da5e730b7e2ecad9ed52812ccae991915050a
Sep 15 07:18:34.210922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.210896    2574 apiserver.go:52] "Watching apiserver"
Sep 15 07:18:34.218621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.218599    2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 07:18:34.219743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.219713    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-24824","openshift-network-operator/iptables-alerter-s68bc","openshift-ovn-kubernetes/ovnkube-node-m99lz","kube-system/global-pull-secret-syncer-6rthp","kube-system/konnectivity-agent-b8f8q","kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd","openshift-dns/node-resolver-84rbg","openshift-multus/multus-additional-cni-plugins-tqxwj","openshift-multus/multus-bn8fl","openshift-multus/network-metrics-daemon-kzwq6","openshift-cluster-node-tuning-operator/tuned-nt989","openshift-image-registry/node-ca-l5q52","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal"]
Sep 15 07:18:34.221594 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.221559    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.223314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.223286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.223970 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.223942    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 07:18:34.224341 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.224072    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.224341 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.224107    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k42fj\""
Sep 15 07:18:34.224341 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.224267    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.224946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.224925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.225877 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.225687    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 07:18:34.225877 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.225690    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-g7dn9\""
Sep 15 07:18:34.225877 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.225773    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.225877 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.225813    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.227587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.227563    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.228747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.227791    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 07:18:34.228747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.227572    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.228747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.228158    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 07:18:34.228747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.228630    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2fvnh\""
Sep 15 07:18:34.229343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.229307    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 07:18:34.230967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.230945    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 07:18:34.231063 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.231015    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.231128 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.231084    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:34.231608 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.231592    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 07:18:34.232714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.232692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.234288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.234262    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.234820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.234800    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-llbfl\""
Sep 15 07:18:34.234913 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.234867    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 07:18:34.235113 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.235093    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 07:18:34.235810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.235790    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.236238 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.236216    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.236775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.236746    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.236881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.236746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rrg7l\""
Sep 15 07:18:34.237329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.237310    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:34.237421 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.237375    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:34.238149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238128    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 07:18:34.238264 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238155    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 07:18:34.238485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238462    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66pt5\""
Sep 15 07:18:34.238610 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238509    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.238610 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238515    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 07:18:34.238955 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238926    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.239055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.238965    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.240905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.240882    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.241115 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.241098    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 07:18:34.241279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.241185    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.241409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.241392    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tk4jp\""
Sep 15 07:18:34.241547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.241526    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.243217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.243141    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.243320 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.243211    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:34.243320 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.243257    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 07:18:34.243536 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.243515    2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 07:18:34.243630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.243523    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dp2bw\""
Sep 15 07:18:34.244915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.244897    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-log-socket\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.244926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-netd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.244954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.244981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovn-node-metrics-cert\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-script-lib\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245032    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245055    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/912a1a00-fcb7-488e-922c-90863e26694b-hosts-file\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-sys-fs\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-kubelet\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245132    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-bin\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245193    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-config\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245238    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7jq\" (UniqueName: \"kubernetes.io/projected/de07649c-6cb2-4423-9fcf-beb23b062f2a-kube-api-access-7d7jq\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-host\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245289    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f46ft\" (UniqueName: \"kubernetes.io/projected/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-kube-api-access-f46ft\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245316    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-slash\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-var-lib-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245379    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245403    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-kubelet-config\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245428    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-agent-certs\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a1a00-fcb7-488e-922c-90863e26694b-tmp-dir\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-device-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245535    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-systemd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-etc-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-dbus\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-registration-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.245714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-etc-selinux\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245878    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd5w6\" (UniqueName: \"kubernetes.io/projected/94dd118d-84c6-404c-aec2-bef2d92d9091-kube-api-access-wd5w6\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.245944    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-systemd-units\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-env-overrides\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246081    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-node-log\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246112    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-konnectivity-ca\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4j2r\" (UniqueName: \"kubernetes.io/projected/912a1a00-fcb7-488e-922c-90863e26694b-kube-api-access-t4j2r\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-socket-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246210    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-serviceca\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-netns\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-ovn\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.246867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.246465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.248651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.248481    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:18:34.248651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.248568    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vvrwr\""
Sep 15 07:18:34.248814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.248678    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 07:18:34.268800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.268766    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9spb"
Sep 15 07:18:34.278174 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.278157    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d9spb"
Sep 15 07:18:34.334306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.334254    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 07:18:34.346773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346752    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-cni-binary-copy\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.346873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346776    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.346873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-socket-dir-parent\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.346873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346808    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-multus\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.346873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346824    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-hostroot\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346867    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-registration-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-etc-selinux\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd5w6\" (UniqueName: \"kubernetes.io/projected/94dd118d-84c6-404c-aec2-bef2d92d9091-kube-api-access-wd5w6\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346929    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-env-overrides\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-run\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346979    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-kubelet\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.346996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347015    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-systemd\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.347050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347039    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-ovn\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347068    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ws8r\" (UniqueName: \"kubernetes.io/projected/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-kube-api-access-5ws8r\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347146    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-kubernetes\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347125    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-registration-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347172    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-etc-tuned\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347170    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-etc-selinux\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-ovn\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-netd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347303    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovn-node-metrics-cert\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347337    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-netd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347366    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347383    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/912a1a00-fcb7-488e-922c-90863e26694b-hosts-file\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.347543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347410    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1224304-cc06-46ab-aa0e-ed1a2a049462-host-slash\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.347438    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/912a1a00-fcb7-488e-922c-90863e26694b-hosts-file\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-system-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.347497    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:18:34.847474851 +0000 UTC m=+2.941641098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-bin\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347541    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-env-overrides\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347543    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-config\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-cni-bin\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347576    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7jq\" (UniqueName: \"kubernetes.io/projected/de07649c-6cb2-4423-9fcf-beb23b062f2a-kube-api-access-7d7jq\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347626    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347647    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-tmp\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347670    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-k8s-cni-cncf-io\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347690    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-bin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-var-lib-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.348314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347756    2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-agent-certs\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cnibin\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347801    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-lib-modules\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347810    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-var-lib-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347825    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-var-lib-kubelet\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-netns\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-systemd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347902    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-etc-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347939    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-systemd\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-dbus\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-etc-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysconfig\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-dbus\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.347999    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1224304-cc06-46ab-aa0e-ed1a2a049462-iptables-alerter-script\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348012    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-config\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348031    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vccv\" (UniqueName: \"kubernetes.io/projected/b1224304-cc06-46ab-aa0e-ed1a2a049462-kube-api-access-6vccv\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.349122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348065    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348089    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-sys\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348148    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-systemd-units\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghb2j\" (UniqueName: \"kubernetes.io/projected/531d1557-eab9-45c5-a333-9122556ce383-kube-api-access-ghb2j\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-systemd-units\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-etc-kubernetes\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348231    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dj99\" (UniqueName: \"kubernetes.io/projected/73b9bc60-993a-40b7-b673-32e1b8e9654b-kube-api-access-9dj99\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348248    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-node-log\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-konnectivity-ca\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4j2r\" (UniqueName: \"kubernetes.io/projected/912a1a00-fcb7-488e-922c-90863e26694b-kube-api-access-t4j2r\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-node-log\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-socket-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-socket-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348486    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-serviceca\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348504    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348523    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-host\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.349786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348548    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-netns\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348582    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-multus-certs\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-log-socket\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-script-lib\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-run-netns\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-log-socket\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-sys-fs\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348711    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-host\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348737    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-system-cni-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfs8m\" (UniqueName: \"kubernetes.io/projected/390df7c3-60f0-41ea-854c-e5003102b214-kube-api-access-cfs8m\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348765    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-konnectivity-ca\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348780    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-sys-fs\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348893    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-host\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-kubelet\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f46ft\" (UniqueName: \"kubernetes.io/projected/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-kube-api-access-f46ft\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.348990    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-conf\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.350317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349007    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-cnibin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-conf-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349056    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-daemon-config\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-slash\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349119    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-kubelet-config\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a1a00-fcb7-488e-922c-90863e26694b-tmp-dir\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349186    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-device-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349192    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-kubelet\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349211    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-modprobe-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349327    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovnkube-script-lib\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-serviceca\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349333    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-host-slash\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349364    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/de07649c-6cb2-4423-9fcf-beb23b062f2a-run-openvswitch\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1d5ee397-08ec-4837-a81f-3e22caddadcc-kubelet-config\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/94dd118d-84c6-404c-aec2-bef2d92d9091-device-dir\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.350809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.349639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/912a1a00-fcb7-488e-922c-90863e26694b-tmp-dir\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.351487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.351225    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/de07649c-6cb2-4423-9fcf-beb23b062f2a-ovn-node-metrics-cert\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.351487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.351318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6af4b9b5-9713-46ae-bfd0-22c7ddf241bb-agent-certs\") pod \"konnectivity-agent-b8f8q\" (UID: \"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb\") " pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.361280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.361256    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f46ft\" (UniqueName: \"kubernetes.io/projected/16ce4021-b6c1-472c-b4fe-379a5d2bf4a6-kube-api-access-f46ft\") pod \"node-ca-l5q52\" (UID: \"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6\") " pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.361390 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.361282    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd5w6\" (UniqueName: \"kubernetes.io/projected/94dd118d-84c6-404c-aec2-bef2d92d9091-kube-api-access-wd5w6\") pod \"aws-ebs-csi-driver-node-2qbhd\" (UID: \"94dd118d-84c6-404c-aec2-bef2d92d9091\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.361997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.361973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4j2r\" (UniqueName: \"kubernetes.io/projected/912a1a00-fcb7-488e-922c-90863e26694b-kube-api-access-t4j2r\") pod \"node-resolver-84rbg\" (UID: \"912a1a00-fcb7-488e-922c-90863e26694b\") " pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.362294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.362274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7jq\" (UniqueName: \"kubernetes.io/projected/de07649c-6cb2-4423-9fcf-beb23b062f2a-kube-api-access-7d7jq\") pod \"ovnkube-node-m99lz\" (UID: \"de07649c-6cb2-4423-9fcf-beb23b062f2a\") " pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.380811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.380760    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal" event={"ID":"193dba6a0f198e81f9ade158f96f4c19","Type":"ContainerStarted","Data":"1e0c6ff669ab4339513d10cfdb395d1f78b6325b9b6642dfcfc75db96593a217"}
Sep 15 07:18:34.382102 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.382078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal" event={"ID":"d9f17e721a455587375069a20c2a5ed0","Type":"ContainerStarted","Data":"fb1ea6457eec340119ee1b32960da5e730b7e2ecad9ed52812ccae991915050a"}
Sep 15 07:18:34.450342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-multus-certs\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450349    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-system-cni-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.450447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450366    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfs8m\" (UniqueName: \"kubernetes.io/projected/390df7c3-60f0-41ea-854c-e5003102b214-kube-api-access-cfs8m\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.450447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450422    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-multus-certs\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450448    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-system-cni-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-conf\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-cnibin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450574    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-conf-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450622    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-daemon-config\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450628    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-cnibin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-conf\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.450663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450653    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-modprobe-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-cni-binary-copy\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-conf-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-socket-dir-parent\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450756    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-multus\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450779    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-hostroot\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450807    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-run\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-modprobe-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450835    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-kubelet\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450861    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450871    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-socket-dir-parent\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450890    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-systemd\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-hostroot\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ws8r\" (UniqueName: \"kubernetes.io/projected/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-kube-api-access-5ws8r\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-kubernetes\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450940    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-multus\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.451123 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.450971    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-etc-tuned\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451009    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451055    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1224304-cc06-46ab-aa0e-ed1a2a049462-host-slash\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451136    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-system-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451181    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-tmp\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-k8s-cni-cncf-io\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451276    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-multus-daemon-config\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-bin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451316    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451334    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/73b9bc60-993a-40b7-b673-32e1b8e9654b-cni-binary-copy\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451362    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cnibin\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451389    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-lib-modules\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451413    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-var-lib-kubelet\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451463    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-cni-bin\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451472    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-var-lib-kubelet\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-netns\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451534    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysconfig\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451556    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-var-lib-kubelet\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451576    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysctl-d\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451624    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-k8s-cni-cncf-io\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cnibin\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-lib-modules\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451716    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-host-run-netns\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.451780    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451783    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b1224304-cc06-46ab-aa0e-ed1a2a049462-host-slash\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.451845    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:34.951825243 +0000 UTC m=+3.045991491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451501    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1224304-cc06-46ab-aa0e-ed1a2a049462-iptables-alerter-script\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-run\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-system-cni-dir\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.452957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451927    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vccv\" (UniqueName: \"kubernetes.io/projected/b1224304-cc06-46ab-aa0e-ed1a2a049462-kube-api-access-6vccv\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451955    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-sysconfig\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.451978    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-sys\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghb2j\" (UniqueName: \"kubernetes.io/projected/531d1557-eab9-45c5-a333-9122556ce383-kube-api-access-ghb2j\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-etc-kubernetes\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452095    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dj99\" (UniqueName: \"kubernetes.io/projected/73b9bc60-993a-40b7-b673-32e1b8e9654b-kube-api-access-9dj99\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452126    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-host\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452061    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b1224304-cc06-46ab-aa0e-ed1a2a049462-iptables-alerter-script\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-sys\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452236    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-host\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452257    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/73b9bc60-993a-40b7-b673-32e1b8e9654b-etc-kubernetes\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452281    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-systemd\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/390df7c3-60f0-41ea-854c-e5003102b214-etc-kubernetes\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.452610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.453623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.453513    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-etc-tuned\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.454293 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.454265    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/390df7c3-60f0-41ea-854c-e5003102b214-tmp\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.460576 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.460555    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:34.460662 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.460581    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:34.460662 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.460595    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:34.460734 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.460662    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:34.960643011 +0000 UTC m=+3.054809271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:34.462971 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.462858    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfs8m\" (UniqueName: \"kubernetes.io/projected/390df7c3-60f0-41ea-854c-e5003102b214-kube-api-access-cfs8m\") pod \"tuned-nt989\" (UID: \"390df7c3-60f0-41ea-854c-e5003102b214\") " pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.464418 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.464369    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dj99\" (UniqueName: \"kubernetes.io/projected/73b9bc60-993a-40b7-b673-32e1b8e9654b-kube-api-access-9dj99\") pod \"multus-bn8fl\" (UID: \"73b9bc60-993a-40b7-b673-32e1b8e9654b\") " pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.464551 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.464516    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghb2j\" (UniqueName: \"kubernetes.io/projected/531d1557-eab9-45c5-a333-9122556ce383-kube-api-access-ghb2j\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.464704 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.464652    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vccv\" (UniqueName: \"kubernetes.io/projected/b1224304-cc06-46ab-aa0e-ed1a2a049462-kube-api-access-6vccv\") pod \"iptables-alerter-s68bc\" (UID: \"b1224304-cc06-46ab-aa0e-ed1a2a049462\") " pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.465000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.464976    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ws8r\" (UniqueName: \"kubernetes.io/projected/2c08a0ed-45d4-47f6-b4bb-432e62eea5e1-kube-api-access-5ws8r\") pod \"multus-additional-cni-plugins-tqxwj\" (UID: \"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1\") " pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.536179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.536153    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd"
Sep 15 07:18:34.544488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.544467    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l5q52"
Sep 15 07:18:34.554550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.554530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:34.560886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.560863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:34.567964 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.567939    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-84rbg"
Sep 15 07:18:34.575802 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.575782    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-tqxwj"
Sep 15 07:18:34.583663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.583645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-s68bc"
Sep 15 07:18:34.590362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.590341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bn8fl"
Sep 15 07:18:34.596225 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.596208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nt989"
Sep 15 07:18:34.777680 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.777653    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16ce4021_b6c1_472c_b4fe_379a5d2bf4a6.slice/crio-956a6f631ab0b548c9ec1ab1670cbbdfc04bce295aa77f9663ed005d74f55437 WatchSource:0}: Error finding container 956a6f631ab0b548c9ec1ab1670cbbdfc04bce295aa77f9663ed005d74f55437: Status 404 returned error can't find the container with id 956a6f631ab0b548c9ec1ab1670cbbdfc04bce295aa77f9663ed005d74f55437
Sep 15 07:18:34.792550 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.792523    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912a1a00_fcb7_488e_922c_90863e26694b.slice/crio-426c3e55b246b8d6ba7611fa7f37f7244fcfa8a7eef7ee2c347fc5b4254b963c WatchSource:0}: Error finding container 426c3e55b246b8d6ba7611fa7f37f7244fcfa8a7eef7ee2c347fc5b4254b963c: Status 404 returned error can't find the container with id 426c3e55b246b8d6ba7611fa7f37f7244fcfa8a7eef7ee2c347fc5b4254b963c
Sep 15 07:18:34.793169 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.793081    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c08a0ed_45d4_47f6_b4bb_432e62eea5e1.slice/crio-cc3de807f4aa97252cbff719f5a8d006abb7f5c4f73a216052d82fec45eb7785 WatchSource:0}: Error finding container cc3de807f4aa97252cbff719f5a8d006abb7f5c4f73a216052d82fec45eb7785: Status 404 returned error can't find the container with id cc3de807f4aa97252cbff719f5a8d006abb7f5c4f73a216052d82fec45eb7785
Sep 15 07:18:34.797320 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.797295    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b9bc60_993a_40b7_b673_32e1b8e9654b.slice/crio-4486875f04fdce416ec07f4d4c4ff46b48be7434a8dbfeee2ab8375fb0a4dd7f WatchSource:0}: Error finding container 4486875f04fdce416ec07f4d4c4ff46b48be7434a8dbfeee2ab8375fb0a4dd7f: Status 404 returned error can't find the container with id 4486875f04fdce416ec07f4d4c4ff46b48be7434a8dbfeee2ab8375fb0a4dd7f
Sep 15 07:18:34.798430 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.798408    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde07649c_6cb2_4423_9fcf_beb23b062f2a.slice/crio-c2c8922561aa761f62023b3125c7ff26f34a4734f31b6900ad91b74c375b523c WatchSource:0}: Error finding container c2c8922561aa761f62023b3125c7ff26f34a4734f31b6900ad91b74c375b523c: Status 404 returned error can't find the container with id c2c8922561aa761f62023b3125c7ff26f34a4734f31b6900ad91b74c375b523c
Sep 15 07:18:34.798761 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.798743    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94dd118d_84c6_404c_aec2_bef2d92d9091.slice/crio-2b66b631425a99cb094978f85da4b96e1c4e2c25d85d651224d5b5e4c023ed73 WatchSource:0}: Error finding container 2b66b631425a99cb094978f85da4b96e1c4e2c25d85d651224d5b5e4c023ed73: Status 404 returned error can't find the container with id 2b66b631425a99cb094978f85da4b96e1c4e2c25d85d651224d5b5e4c023ed73
Sep 15 07:18:34.800373 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.800341    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1224304_cc06_46ab_aa0e_ed1a2a049462.slice/crio-21122da236b86866c4c38c6c28e464e7a34f893adcd6a13acb3f7bff3dc03856 WatchSource:0}: Error finding container 21122da236b86866c4c38c6c28e464e7a34f893adcd6a13acb3f7bff3dc03856: Status 404 returned error can't find the container with id 21122da236b86866c4c38c6c28e464e7a34f893adcd6a13acb3f7bff3dc03856
Sep 15 07:18:34.802753 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.801359    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod390df7c3_60f0_41ea_854c_e5003102b214.slice/crio-57eac8868f24def7fd5845d5d3dfef8718421dd91743b0e1073fc8e8a2373e88 WatchSource:0}: Error finding container 57eac8868f24def7fd5845d5d3dfef8718421dd91743b0e1073fc8e8a2373e88: Status 404 returned error can't find the container with id 57eac8868f24def7fd5845d5d3dfef8718421dd91743b0e1073fc8e8a2373e88
Sep 15 07:18:34.802753 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:18:34.802343    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6af4b9b5_9713_46ae_bfd0_22c7ddf241bb.slice/crio-81376906f5011b9260d5b707b91f5db8235ebbb7918db124e3563e04de9f897c WatchSource:0}: Error finding container 81376906f5011b9260d5b707b91f5db8235ebbb7918db124e3563e04de9f897c: Status 404 returned error can't find the container with id 81376906f5011b9260d5b707b91f5db8235ebbb7918db124e3563e04de9f897c
Sep 15 07:18:34.855559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.855506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:34.855647 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.855632    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:34.855720 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.855707    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:18:35.855685067 +0000 UTC m=+3.949851331 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:34.956146 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:34.956107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:34.956276 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.956249    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:34.956346 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:34.956309    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:35.956288052 +0000 UTC m=+4.050454312 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:35.057627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.057318    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:35.057627 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.057487    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:35.057798 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.057631    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:35.057798 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.057648    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:35.057798 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.057713    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:36.057694045 +0000 UTC m=+4.151860541 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:35.280768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.280717    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 07:13:34 +0000 UTC" deadline="2028-05-05 12:50:32.086069434 +0000 UTC"
Sep 15 07:18:35.280768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.280763    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14357h31m56.805308838s"
Sep 15 07:18:35.372363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.372254    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:35.372540 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.372385    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:35.386095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.386059    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l5q52" event={"ID":"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6","Type":"ContainerStarted","Data":"956a6f631ab0b548c9ec1ab1670cbbdfc04bce295aa77f9663ed005d74f55437"}
Sep 15 07:18:35.389789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.389190    2574 generic.go:356] "Generic (PLEG): container finished" podID="193dba6a0f198e81f9ade158f96f4c19" containerID="6cd8e8f386e96d5479b35e53a81093b916e3b12a4bc06d076fe41e8b2d5dc1ff" exitCode=0
Sep 15 07:18:35.389789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.389773    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal" event={"ID":"193dba6a0f198e81f9ade158f96f4c19","Type":"ContainerDied","Data":"6cd8e8f386e96d5479b35e53a81093b916e3b12a4bc06d076fe41e8b2d5dc1ff"}
Sep 15 07:18:35.394414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.394380    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s68bc" event={"ID":"b1224304-cc06-46ab-aa0e-ed1a2a049462","Type":"ContainerStarted","Data":"21122da236b86866c4c38c6c28e464e7a34f893adcd6a13acb3f7bff3dc03856"}
Sep 15 07:18:35.396239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.396208    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd" event={"ID":"94dd118d-84c6-404c-aec2-bef2d92d9091","Type":"ContainerStarted","Data":"2b66b631425a99cb094978f85da4b96e1c4e2c25d85d651224d5b5e4c023ed73"}
Sep 15 07:18:35.398175 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.398127    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bn8fl" event={"ID":"73b9bc60-993a-40b7-b673-32e1b8e9654b","Type":"ContainerStarted","Data":"4486875f04fdce416ec07f4d4c4ff46b48be7434a8dbfeee2ab8375fb0a4dd7f"}
Sep 15 07:18:35.403309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.403283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84rbg" event={"ID":"912a1a00-fcb7-488e-922c-90863e26694b","Type":"ContainerStarted","Data":"426c3e55b246b8d6ba7611fa7f37f7244fcfa8a7eef7ee2c347fc5b4254b963c"}
Sep 15 07:18:35.405054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.405030    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b8f8q" event={"ID":"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb","Type":"ContainerStarted","Data":"81376906f5011b9260d5b707b91f5db8235ebbb7918db124e3563e04de9f897c"}
Sep 15 07:18:35.409637 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.409566    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nt989" event={"ID":"390df7c3-60f0-41ea-854c-e5003102b214","Type":"ContainerStarted","Data":"57eac8868f24def7fd5845d5d3dfef8718421dd91743b0e1073fc8e8a2373e88"}
Sep 15 07:18:35.411186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.411132    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"c2c8922561aa761f62023b3125c7ff26f34a4734f31b6900ad91b74c375b523c"}
Sep 15 07:18:35.413580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.412792    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerStarted","Data":"cc3de807f4aa97252cbff719f5a8d006abb7f5c4f73a216052d82fec45eb7785"}
Sep 15 07:18:35.866470 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.866369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:35.866660 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.866575    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:35.866660 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.866642    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:18:37.866622721 +0000 UTC m=+5.960788965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:35.967840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:35.967802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:35.968044 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.967955    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:35.968044 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:35.968018    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:37.967998442 +0000 UTC m=+6.062164687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:36.069260 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:36.068526    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:36.069260 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.068755    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:36.069260 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.068776    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:36.069260 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.068790    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:36.069260 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.068849    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:38.068831643 +0000 UTC m=+6.162997919 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:36.281898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:36.281797    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 07:13:34 +0000 UTC" deadline="2028-06-30 19:35:27.404573204 +0000 UTC"
Sep 15 07:18:36.281898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:36.281838    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15708h16m51.122740073s"
Sep 15 07:18:36.373807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:36.373721    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:36.373976 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.373866    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:36.373976 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:36.373915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:36.374095 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:36.374019    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:37.372727 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:37.372673    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:37.373354 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:37.372804    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:37.888766 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:37.888731    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:37.888952 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:37.888914    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:37.889015 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:37.888986    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:18:41.88896513 +0000 UTC m=+9.983131378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:37.989819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:37.989781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:37.990010 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:37.989960    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:37.990079 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:37.990022    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:41.990002435 +0000 UTC m=+10.084168692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:38.091564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:38.090759    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:38.091564 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.090951    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:38.091564 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.090975    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:38.091564 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.090991    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:38.091564 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.091047    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:42.091027612 +0000 UTC m=+10.185193856 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:38.377145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:38.373862    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:38.377145 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.374010    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:38.377145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:38.374429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:38.377145 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:38.374531    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:39.372628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:39.372597    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:39.372806 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:39.372722    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:40.376538 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:40.376509    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:40.377000 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:40.376656    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:40.377079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:40.377043    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:40.377176 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:40.377153    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:41.372874 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:41.372533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:41.372874 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:41.372679    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:41.924390 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:41.924268    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:41.924923 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:41.924425    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:41.924923 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:41.924479    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:18:49.924464436 +0000 UTC m=+18.018630676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:42.025006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:42.024974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:42.025185 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.025150    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:42.025264 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.025233    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:50.02521443 +0000 UTC m=+18.119380689 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:42.125814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:42.125709    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:42.125976 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.125860    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:42.125976 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.125884    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:42.125976 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.125898    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:42.125976 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.125954    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:18:50.125934274 +0000 UTC m=+18.220100532 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:42.373263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:42.373169    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:42.373405 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.373290    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:42.373615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:42.373518    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:42.373716 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:42.373636    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:43.372302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:43.372268    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:43.372755 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:43.372386    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:44.372243 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:44.372209    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:44.372389 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:44.372333    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:44.372706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:44.372446    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:44.372706 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:44.372547    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:45.372236 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:45.372206    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:45.372409 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:45.372331    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:46.372604 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:46.372576    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:46.373063 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:46.372710    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:46.373063 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:46.372717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:46.373063 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:46.372801    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:47.372265 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:47.372230    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:47.372425 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:47.372350    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:48.372067 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:48.372035    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:48.372641 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:48.372167    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:48.372641 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:48.372226    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:48.372641 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:48.372339    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:49.372644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:49.372603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:49.373217 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:49.372726    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:49.986220 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:49.986189    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:49.986392 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:49.986343    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:49.986438 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:49.986410    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:19:05.986393727 +0000 UTC m=+34.080559967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:18:50.086672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:50.086640    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:50.086845 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.086801    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:50.086911 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.086874    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:06.086853481 +0000 UTC m=+34.181019724 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:18:50.187598 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:50.187567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:50.187753 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.187735    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:18:50.187813 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.187756    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:18:50.187813 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.187766    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:50.187891 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.187821    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:06.187803473 +0000 UTC m=+34.281969728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:18:50.372455 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:50.372383    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:50.372606 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.372504    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:50.372606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:50.372587    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:50.372975 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:50.372691    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:51.372887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:51.372850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:51.373345 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:51.372970    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:52.372884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:52.372850    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:52.373057 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:52.372966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:52.373057 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:52.373024    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:52.373502 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:52.373138    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:53.372521 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.372249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:53.372645 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:53.372623    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:53.460572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.460547    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd" event={"ID":"94dd118d-84c6-404c-aec2-bef2d92d9091","Type":"ContainerStarted","Data":"7536a2f4deec41f7213bc7223adafec553279f4a73e80e6134af2aed8425399b"}
Sep 15 07:18:53.461960 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.461938    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bn8fl" event={"ID":"73b9bc60-993a-40b7-b673-32e1b8e9654b","Type":"ContainerStarted","Data":"c8108158e863e7f9f0573f8e5a640b4d9b7e99a59bb526831b8d91e14a46243f"}
Sep 15 07:18:53.463327 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.463305    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-84rbg" event={"ID":"912a1a00-fcb7-488e-922c-90863e26694b","Type":"ContainerStarted","Data":"11a583bbf6a13fc1e6029fba2833bad50e34031d6cf9114d1b51e45d4995f96b"}
Sep 15 07:18:53.464485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.464466    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-b8f8q" event={"ID":"6af4b9b5-9713-46ae-bfd0-22c7ddf241bb","Type":"ContainerStarted","Data":"5a6ece63b559aba63eb8671ac24e1c6f329b9c327ad20eb5ff5043006abed938"}
Sep 15 07:18:53.465723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.465701    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nt989" event={"ID":"390df7c3-60f0-41ea-854c-e5003102b214","Type":"ContainerStarted","Data":"0107dcdf791b0fed5a8d644b4e66e31b7acb88a1a178d28e9cd8f480ae80c9b3"}
Sep 15 07:18:53.468085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.468067    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"fed6e6946b7fe85ab6a3d0fb4d1ce28c9aab73a4d6b3a47c34abbd87ee1e1b36"}
Sep 15 07:18:53.468209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.468143    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"fcbaa30ecddfa4b918303537ef1afdc47e5e371737df1b53ae33d14cd2a22a76"}
Sep 15 07:18:53.468209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.468173    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"9c343f4b67308a0f2bc526a27b4d5ac88efc5e3cfd5c0a9ab46ea520fe7985ce"}
Sep 15 07:18:53.468209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.468202    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"93fbe6abbce945cf0cb5c8db455e562ddfcc074838aaa255da0c05c5e6b38ccd"}
Sep 15 07:18:53.468347 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.468236    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"5afc0260615a0215003829160f7d65c435aa8ca18c1575bf046f09f704fee77c"}
Sep 15 07:18:53.469503 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.469480    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="23c083ea98af11cb47acfb779170ab371e6cb1f86f15ebba82cdfa0138969540" exitCode=0
Sep 15 07:18:53.469605 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.469545    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"23c083ea98af11cb47acfb779170ab371e6cb1f86f15ebba82cdfa0138969540"}
Sep 15 07:18:53.471075 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.471050    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l5q52" event={"ID":"16ce4021-b6c1-472c-b4fe-379a5d2bf4a6","Type":"ContainerStarted","Data":"2f8857a31beed1975e12d335212030509cb684ca3ae45c4c41908fe3b39a2715"}
Sep 15 07:18:53.472538 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.472512    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal" event={"ID":"d9f17e721a455587375069a20c2a5ed0","Type":"ContainerStarted","Data":"34242e62e58641adc7e93894eb64400f6458afb1d157d24bc7375cbb2141dbf1"}
Sep 15 07:18:53.474302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.474283    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal" event={"ID":"193dba6a0f198e81f9ade158f96f4c19","Type":"ContainerStarted","Data":"cf5dfd344ed6df52c7d7053187947c2800da5f20492a8f0c79e54683a018f4a3"}
Sep 15 07:18:53.525900 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.525856    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-84rbg" podStartSLOduration=3.449288863 podStartE2EDuration="21.52584126s" totalImagesPullingTime="18.076552397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.508646524 +0000 UTC m=+21.602812786" watchObservedRunningTime="2026-09-15 07:18:53.52584126 +0000 UTC m=+21.620007541"
Sep 15 07:18:53.526003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.525984    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bn8fl" podStartSLOduration=3.329479512 podStartE2EDuration="21.52597987s" totalImagesPullingTime="18.196500358s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.525759092 +0000 UTC m=+21.619925381" watchObservedRunningTime="2026-09-15 07:18:53.52597987 +0000 UTC m=+21.620146131"
Sep 15 07:18:53.539834 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.539799    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-227.ec2.internal" podStartSLOduration=19.539785838 podStartE2EDuration="19.539785838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.53970569 +0000 UTC m=+21.633871957" watchObservedRunningTime="2026-09-15 07:18:53.539785838 +0000 UTC m=+21.633952124"
Sep 15 07:18:53.570020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.569983    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-227.ec2.internal" podStartSLOduration=19.569972358 podStartE2EDuration="19.569972358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.554265965 +0000 UTC m=+21.648432227" watchObservedRunningTime="2026-09-15 07:18:53.569972358 +0000 UTC m=+21.664138641"
Sep 15 07:18:53.570325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.570300    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l5q52" podStartSLOduration=3.570721712 podStartE2EDuration="21.570292519s" totalImagesPullingTime="17.999570807s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.569145111 +0000 UTC m=+21.663311373" watchObservedRunningTime="2026-09-15 07:18:53.570292519 +0000 UTC m=+21.664458780"
Sep 15 07:18:53.583806 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.583765    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-b8f8q" podStartSLOduration=3.5204128519999998 podStartE2EDuration="21.583747601s" totalImagesPullingTime="18.063334749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.58314409 +0000 UTC m=+21.677310351" watchObservedRunningTime="2026-09-15 07:18:53.583747601 +0000 UTC m=+21.677913864"
Sep 15 07:18:53.605032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.604996    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nt989" podStartSLOduration=3.543960375 podStartE2EDuration="21.60498668s" totalImagesPullingTime="18.061026305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:53.604732709 +0000 UTC m=+21.698898974" watchObservedRunningTime="2026-09-15 07:18:53.60498668 +0000 UTC m=+21.699152942"
Sep 15 07:18:53.669641 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.669590    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:53.670012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.669997    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:53.683771 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.683747    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:53.754326 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:53.754304    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:54.372575 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.372473    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:54.372715 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:54.372588    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:54.372715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.372692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:54.372823 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:54.372806    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:54.479649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.479616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"7914aed552d2cacc555001148a77678a4bc0cbdc5f97c6785768956b78bbf585"}
Sep 15 07:18:54.481478 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.481333    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-s68bc" event={"ID":"b1224304-cc06-46ab-aa0e-ed1a2a049462","Type":"ContainerStarted","Data":"5027c84d0db6797ceff8412e4df9db30d2bf4d1d8f2e2b5b1c2b16588b94f899"}
Sep 15 07:18:54.482533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.482512    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:54.482649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.482569    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-b8f8q"
Sep 15 07:18:54.508087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.508048    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-s68bc" podStartSLOduration=4.465584662 podStartE2EDuration="22.508035824s" totalImagesPullingTime="18.042451162s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:54.495886287 +0000 UTC m=+22.590052548" watchObservedRunningTime="2026-09-15 07:18:54.508035824 +0000 UTC m=+22.602202086"
Sep 15 07:18:54.617205 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.617184    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:54.679495 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.679433    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:54.833362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.833341    2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 07:18:54.905460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:54.905423    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:55.308943 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.308828    2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T07:18:54.833357202Z","UUID":"88390eaa-20f2-4c66-b626-f0483ebcc5f7","Handler":null,"Name":"","Endpoint":""}
Sep 15 07:18:55.310190 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.310171    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 07:18:55.310298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.310199    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 07:18:55.324463 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.324441    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:55.372670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.372640    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:55.372816 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:55.372746    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:55.419590 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.419565    2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log" limit=1024
Sep 15 07:18:55.484947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:55.484921    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd" event={"ID":"94dd118d-84c6-404c-aec2-bef2d92d9091","Type":"ContainerStarted","Data":"163bb46ec2ac9e204569bd3ea92e8c043c74ed91b6bc17d2dc94174e7766225a"}
Sep 15 07:18:56.372160 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:56.372070    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:56.372270 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:56.372194    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:56.372270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:56.372240    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:56.372379 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:56.372356    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:56.488747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:56.488673    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd" event={"ID":"94dd118d-84c6-404c-aec2-bef2d92d9091","Type":"ContainerStarted","Data":"b286da550b87e0929cc2f68e666a5e319103e54c18deb429f8e147548bcc7658"}
Sep 15 07:18:56.492047 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:56.492025    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"34c8dce2fe6fd5323014579e8bb842d0c514206d0743076b62a00c03cef28f22"}
Sep 15 07:18:56.504643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:56.504582    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2qbhd" podStartSLOduration=4.452512052 podStartE2EDuration="24.504565218s" totalImagesPullingTime="20.052053166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:56.504290078 +0000 UTC m=+24.598456340" watchObservedRunningTime="2026-09-15 07:18:56.504565218 +0000 UTC m=+24.598731483"
Sep 15 07:18:57.372756 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:57.372564    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:57.372896 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:57.372832    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:58.372082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:58.372043    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:18:58.372082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:58.372089    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:18:58.372553 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:58.372201    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:18:58.372553 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:58.372313    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:18:59.372223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.372200    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:18:59.372632 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:18:59.372296    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:18:59.501698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.501491    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" event={"ID":"de07649c-6cb2-4423-9fcf-beb23b062f2a","Type":"ContainerStarted","Data":"86ef6edc63d6db5524680caeb1fe0aa5124cfd19b80baf2f189a72c97a8ddef4"}
Sep 15 07:18:59.501810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.501755    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:59.503555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.503523    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="ac30b15656bd27b27ef9248012a500096a9abb7f4d14f826cc7cbdb92de814e7" exitCode=0
Sep 15 07:18:59.503662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.503570    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"ac30b15656bd27b27ef9248012a500096a9abb7f4d14f826cc7cbdb92de814e7"}
Sep 15 07:18:59.516379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.516362    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:18:59.528550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:18:59.528513    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz" podStartSLOduration=9.459939435 podStartE2EDuration="27.528500232s" totalImagesPullingTime="18.068560797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:18:59.527987729 +0000 UTC m=+27.622153992" watchObservedRunningTime="2026-09-15 07:18:59.528500232 +0000 UTC m=+27.622666494"
Sep 15 07:19:00.372582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.372523    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:00.373177 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:00.372682    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:19:00.373177 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.372692    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:00.373177 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:00.372775    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:19:00.448699 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.448670    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:19:00.506649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.506623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:19:00.521139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.521106    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:19:00.760775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.760621    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rthp"]
Sep 15 07:19:00.760775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.760739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:00.760947 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:00.760829    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:19:00.763680 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.763654    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzwq6"]
Sep 15 07:19:00.763804 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.763763    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:00.763872 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:00.763844    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:19:00.764561 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.764538    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-24824"]
Sep 15 07:19:00.764791 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:00.764634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:00.764791 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:00.764732    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:19:01.510752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:01.510724    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="5f72e6757378132ef4b9749e0be9cdaacfc6fae23bbfe638fa3eb854bc063c0d" exitCode=0
Sep 15 07:19:01.511569 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:01.510815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"5f72e6757378132ef4b9749e0be9cdaacfc6fae23bbfe638fa3eb854bc063c0d"}
Sep 15 07:19:02.375457 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:02.375428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:02.375618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:02.375428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:02.375618 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:02.375543    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:19:02.375618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:02.375429    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:02.375618 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:02.375596    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:19:02.375817 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:02.375677    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:19:03.518277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:03.518000    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="37c936436f6475a1fb4d020f2fd766ec8e67f10b2e9e4e6408b415b135a77426" exitCode=0
Sep 15 07:19:03.518836 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:03.518294    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"37c936436f6475a1fb4d020f2fd766ec8e67f10b2e9e4e6408b415b135a77426"}
Sep 15 07:19:04.376130 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:04.376084    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:04.376274 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:04.376090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:04.376274 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:04.376090    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:04.376274 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:04.376206    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6rthp" podUID="1d5ee397-08ec-4837-a81f-3e22caddadcc"
Sep 15 07:19:04.376447 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:04.376291    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-24824" podUID="af05362c-5502-4642-8611-dca76ad8c1f7"
Sep 15 07:19:04.376447 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:04.376381    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:19:05.670423 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.670350    2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-227.ec2.internal" event="NodeReady"
Sep 15 07:19:05.670868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.670458    2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 07:19:05.721219 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.721195    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dt7v6"]
Sep 15 07:19:05.723279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.723260    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:05.725391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.725367    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 07:19:05.725503 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.725409    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6vjvf"]
Sep 15 07:19:05.725503 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.725424    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfj9f\""
Sep 15 07:19:05.725650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.725635    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 07:19:05.729531 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.729514    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:05.731502 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.731481    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 07:19:05.731633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.731540    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 07:19:05.731633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.731577    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpxmd\""
Sep 15 07:19:05.731749 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.731673    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 07:19:05.734079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.734059    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dt7v6"]
Sep 15 07:19:05.737751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.737731    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vjvf"]
Sep 15 07:19:05.910664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ca0b761-ffd4-409c-bcbd-379522595180-tmp-dir\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:05.910835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910689    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqht5\" (UniqueName: \"kubernetes.io/projected/7ca0b761-ffd4-409c-bcbd-379522595180-kube-api-access-fqht5\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:05.910835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910715    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-cert\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:05.910835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5nz\" (UniqueName: \"kubernetes.io/projected/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-kube-api-access-ft5nz\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:05.910835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910786    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:05.910972 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:05.910852    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca0b761-ffd4-409c-bcbd-379522595180-config-volume\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.011946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.011882    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5nz\" (UniqueName: \"kubernetes.io/projected/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-kube-api-access-ft5nz\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:06.011946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.011922    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.012109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.011951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca0b761-ffd4-409c-bcbd-379522595180-config-volume\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.012109 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.012038    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:06.012109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012059    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ca0b761-ffd4-409c-bcbd-379522595180-tmp-dir\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.012109 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.012099    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:06.512078733 +0000 UTC m=+34.606244977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:06.012340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqht5\" (UniqueName: \"kubernetes.io/projected/7ca0b761-ffd4-409c-bcbd-379522595180-kube-api-access-fqht5\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.012340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-cert\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:06.012340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:06.012487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ca0b761-ffd4-409c-bcbd-379522595180-tmp-dir\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.012487 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.012426    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 07:19:06.012487 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.012479    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret podName:1d5ee397-08ec-4837-a81f-3e22caddadcc nodeName:}" failed. No retries permitted until 2026-09-15 07:19:38.012461517 +0000 UTC m=+66.106627791 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret") pod "global-pull-secret-syncer-6rthp" (UID: "1d5ee397-08ec-4837-a81f-3e22caddadcc") : object "kube-system"/"original-pull-secret" not registered
Sep 15 07:19:06.012617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.012547    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca0b761-ffd4-409c-bcbd-379522595180-config-volume\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.016553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.016537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-cert\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:06.021743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.021644    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5nz\" (UniqueName: \"kubernetes.io/projected/964bc816-53aa-4e24-bc1c-af2d4dfdf27a-kube-api-access-ft5nz\") pod \"ingress-canary-6vjvf\" (UID: \"964bc816-53aa-4e24-bc1c-af2d4dfdf27a\") " pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:06.021743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.021727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqht5\" (UniqueName: \"kubernetes.io/projected/7ca0b761-ffd4-409c-bcbd-379522595180-kube-api-access-fqht5\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.041929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.041905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6vjvf"
Sep 15 07:19:06.113055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.113021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:06.113264 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.113182    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:19:06.113264 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.113253    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:38.113233362 +0000 UTC m=+66.207399608 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 07:19:06.213812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.213788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:06.213990 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.213972    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 07:19:06.214053 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.213997    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 07:19:06.214053 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.214008    2574 projected.go:196] Error preparing data for projected volume kube-api-access-br6qb for pod openshift-network-diagnostics/network-check-target-24824: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:19:06.214053 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.214052    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb podName:af05362c-5502-4642-8611-dca76ad8c1f7 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:38.214040209 +0000 UTC m=+66.308206449 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-br6qb" (UniqueName: "kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb") pod "network-check-target-24824" (UID: "af05362c-5502-4642-8611-dca76ad8c1f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 07:19:06.314734 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.314667    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6vjvf"]
Sep 15 07:19:06.318022 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:19:06.317999    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964bc816_53aa_4e24_bc1c_af2d4dfdf27a.slice/crio-b07b28660b6f103309898726c1806d6070f3a7886261fac550c7131c60f342cb WatchSource:0}: Error finding container b07b28660b6f103309898726c1806d6070f3a7886261fac550c7131c60f342cb: Status 404 returned error can't find the container with id b07b28660b6f103309898726c1806d6070f3a7886261fac550c7131c60f342cb
Sep 15 07:19:06.372547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.372522    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:06.372646 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.372533    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:06.372705 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.372661    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:06.375000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.374978    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 07:19:06.375098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.375011    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ftj64\""
Sep 15 07:19:06.375098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.374983    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 07:19:06.375230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.375018    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 07:19:06.375547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.375529    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d7gwj\""
Sep 15 07:19:06.375628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.375537    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 07:19:06.516498 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.516471    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:06.516601 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.516567    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:06.516642 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:06.516625    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:07.516612515 +0000 UTC m=+35.610778756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:06.525760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:06.525739    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vjvf" event={"ID":"964bc816-53aa-4e24-bc1c-af2d4dfdf27a","Type":"ContainerStarted","Data":"b07b28660b6f103309898726c1806d6070f3a7886261fac550c7131c60f342cb"}
Sep 15 07:19:07.523388 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:07.523293    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:07.523823 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:07.523458    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:07.523823 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:07.523526    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:09.523509455 +0000 UTC m=+37.617675718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:09.538187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:09.537983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:09.538616 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:09.538152    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:09.538616 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:09.538334    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:13.538313892 +0000 UTC m=+41.632480139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:10.537796 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:10.537725    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="91182550ef9f1d733a61782a4a48db10ba1a71fe8d9f050b50a2ae4c525907a9" exitCode=0
Sep 15 07:19:10.537945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:10.537809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"91182550ef9f1d733a61782a4a48db10ba1a71fe8d9f050b50a2ae4c525907a9"}
Sep 15 07:19:10.539290 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:10.539268    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6vjvf" event={"ID":"964bc816-53aa-4e24-bc1c-af2d4dfdf27a","Type":"ContainerStarted","Data":"e67e09ea831ab1989abe82d055b6b75ef530315de83a16d2a0f265ed44a94381"}
Sep 15 07:19:10.570053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:10.570005    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6vjvf" podStartSLOduration=1.752469564 podStartE2EDuration="5.56998771s" totalImagesPullingTime="3.817518146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:10.569985152 +0000 UTC m=+38.664151414" watchObservedRunningTime="2026-09-15 07:19:10.56998771 +0000 UTC m=+38.664153973"
Sep 15 07:19:11.544517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:11.544488    2574 generic.go:356] "Generic (PLEG): container finished" podID="2c08a0ed-45d4-47f6-b4bb-432e62eea5e1" containerID="cb41fa57559061897a45bba3accb202a5c03c696d14b49b3b128cf1477f50d78" exitCode=0
Sep 15 07:19:11.544883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:11.544577    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerDied","Data":"cb41fa57559061897a45bba3accb202a5c03c696d14b49b3b128cf1477f50d78"}
Sep 15 07:19:12.549834 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:12.549785    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" event={"ID":"2c08a0ed-45d4-47f6-b4bb-432e62eea5e1","Type":"ContainerStarted","Data":"269aac03a4e72ab1e29a46b46187670029fe2a8d156a3676b945762351b79499"}
Sep 15 07:19:12.597443 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:12.597392    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-tqxwj" podStartSLOduration=7.907061459 podStartE2EDuration="40.597377829s" totalImagesPullingTime="32.69031637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:12.596737811 +0000 UTC m=+40.690904089" watchObservedRunningTime="2026-09-15 07:19:12.597377829 +0000 UTC m=+40.691544092"
Sep 15 07:19:13.566867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:13.566827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:13.567259 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:13.566932    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:13.567259 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:13.566991    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:21.566972187 +0000 UTC m=+49.661138432 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:21.623289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:21.623251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:21.623856 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:21.623437    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:21.623856 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:21.623525    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:19:37.623501895 +0000 UTC m=+65.717668183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:32.522451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:32.522426    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m99lz"
Sep 15 07:19:33.568014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.567985    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"]
Sep 15 07:19:33.570555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.570538    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.574214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.574187    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 07:19:33.574336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.574196    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 07:19:33.574336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.574286    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 07:19:33.574462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.574448    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 07:19:33.593776 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.593755    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"]
Sep 15 07:19:33.595765 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.595748    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.596809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.596789    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"]
Sep 15 07:19:33.598536 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.598518    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 07:19:33.598668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.598652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 07:19:33.598735 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.598680    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 07:19:33.598979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.598964    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 07:19:33.603715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.603696    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-tmp\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.603788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.603729    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6f82\" (UniqueName: \"kubernetes.io/projected/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-kube-api-access-v6f82\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.603837 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.603821    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-klusterlet-config\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.623990 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.623973    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"]
Sep 15 07:19:33.704335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704339    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-tmp\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.704481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704464    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6f82\" (UniqueName: \"kubernetes.io/projected/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-kube-api-access-v6f82\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.704549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704486    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aded570-6864-4ffa-abc1-f6d529e8c81b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rb7\" (UniqueName: \"kubernetes.io/projected/2aded570-6864-4ffa-abc1-f6d529e8c81b-kube-api-access-n7rb7\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.704648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-klusterlet-config\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.704781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.704762    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-tmp\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.707721 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.707695    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-klusterlet-config\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.714020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.713997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6f82\" (UniqueName: \"kubernetes.io/projected/48b8e7c5-6f1c-4c91-9546-1f8ed041daff-kube-api-access-v6f82\") pod \"klusterlet-addon-workmgr-c985588b4-46r6z\" (UID: \"48b8e7c5-6f1c-4c91-9546-1f8ed041daff\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.805555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.805643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.805643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.805643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805635    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aded570-6864-4ffa-abc1-f6d529e8c81b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.805784 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805664    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.805784 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.805679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rb7\" (UniqueName: \"kubernetes.io/projected/2aded570-6864-4ffa-abc1-f6d529e8c81b-kube-api-access-n7rb7\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.806352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.806324    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aded570-6864-4ffa-abc1-f6d529e8c81b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.808039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.808015    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.808143 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.808022    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.808323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.808301    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-hub\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.808396 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.808375    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/2aded570-6864-4ffa-abc1-f6d529e8c81b-ca\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.815651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.815630    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rb7\" (UniqueName: \"kubernetes.io/projected/2aded570-6864-4ffa-abc1-f6d529e8c81b-kube-api-access-n7rb7\") pod \"cluster-proxy-proxy-agent-675bcfb866-kt4mb\" (UID: \"2aded570-6864-4ffa-abc1-f6d529e8c81b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:33.878972 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.878928    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:33.915739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:33.915711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:19:34.145491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:34.145434    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"]
Sep 15 07:19:34.148162 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:19:34.148136    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b8e7c5_6f1c_4c91_9546_1f8ed041daff.slice/crio-304c1af531938e8a0c1eb0e931424083f2a7dfb79cd25047115a823106aeb5c5 WatchSource:0}: Error finding container 304c1af531938e8a0c1eb0e931424083f2a7dfb79cd25047115a823106aeb5c5: Status 404 returned error can't find the container with id 304c1af531938e8a0c1eb0e931424083f2a7dfb79cd25047115a823106aeb5c5
Sep 15 07:19:34.192617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:34.192597    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"]
Sep 15 07:19:34.195192 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:19:34.195170    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2aded570_6864_4ffa_abc1_f6d529e8c81b.slice/crio-c4febb14c98b183d0fb11898a94e24696ee729a3c96051bf9c6de5dc1af9738c WatchSource:0}: Error finding container c4febb14c98b183d0fb11898a94e24696ee729a3c96051bf9c6de5dc1af9738c: Status 404 returned error can't find the container with id c4febb14c98b183d0fb11898a94e24696ee729a3c96051bf9c6de5dc1af9738c
Sep 15 07:19:34.601299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:34.601268    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerStarted","Data":"c4febb14c98b183d0fb11898a94e24696ee729a3c96051bf9c6de5dc1af9738c"}
Sep 15 07:19:34.602420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:34.602399    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z" event={"ID":"48b8e7c5-6f1c-4c91-9546-1f8ed041daff","Type":"ContainerStarted","Data":"304c1af531938e8a0c1eb0e931424083f2a7dfb79cd25047115a823106aeb5c5"}
Sep 15 07:19:37.633435 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:37.633406    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:19:37.633862 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:37.633588    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:19:37.633862 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:37.633665    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:20:09.633645887 +0000 UTC m=+97.727812132 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:19:38.037724 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.037686    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:38.040154 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.040129    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 07:19:38.051300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.051277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1d5ee397-08ec-4837-a81f-3e22caddadcc-original-pull-secret\") pod \"global-pull-secret-syncer-6rthp\" (UID: \"1d5ee397-08ec-4837-a81f-3e22caddadcc\") " pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:38.138337 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.138305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:19:38.140440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.140417    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 07:19:38.149308 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:38.149286    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:19:38.149422 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:19:38.149358    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:20:42.14933672 +0000 UTC m=+130.243502967 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : secret "metrics-daemon-secret" not found
Sep 15 07:19:38.194165 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.194139    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6rthp"
Sep 15 07:19:38.239649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.239623    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:38.242235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.242212    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 07:19:38.252463 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.252437    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 07:19:38.263243 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.263192    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br6qb\" (UniqueName: \"kubernetes.io/projected/af05362c-5502-4642-8611-dca76ad8c1f7-kube-api-access-br6qb\") pod \"network-check-target-24824\" (UID: \"af05362c-5502-4642-8611-dca76ad8c1f7\") " pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:38.490676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.490652    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-d7gwj\""
Sep 15 07:19:38.498787 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:38.498760    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:40.007099 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.007026    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6rthp"]
Sep 15 07:19:40.010157 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:19:40.010132    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d5ee397_08ec_4837_a81f_3e22caddadcc.slice/crio-569acd8479a3397463dff5ed1a5257a8a2323e56699f00684217d3dd39d200e5 WatchSource:0}: Error finding container 569acd8479a3397463dff5ed1a5257a8a2323e56699f00684217d3dd39d200e5: Status 404 returned error can't find the container with id 569acd8479a3397463dff5ed1a5257a8a2323e56699f00684217d3dd39d200e5
Sep 15 07:19:40.012275 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.012254    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-24824"]
Sep 15 07:19:40.014646 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:19:40.014625    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf05362c_5502_4642_8611_dca76ad8c1f7.slice/crio-78ebe04da2fa719285da356e8516099b563a25323cc7c8159c9625bc750b189e WatchSource:0}: Error finding container 78ebe04da2fa719285da356e8516099b563a25323cc7c8159c9625bc750b189e: Status 404 returned error can't find the container with id 78ebe04da2fa719285da356e8516099b563a25323cc7c8159c9625bc750b189e
Sep 15 07:19:40.624405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.624367    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerStarted","Data":"1e95f99522d600ce98917653d239c05e896cf00e0652d77936637db89b566b64"}
Sep 15 07:19:40.624596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.624422    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerStarted","Data":"add76e1f5e118698ad916fad169f0fbbe7fca3623abb3e3839022d5918879878"}
Sep 15 07:19:40.624596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.624441    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerStarted","Data":"ebb40a3ce662810806df987abed0b99aa1cd7f619b219ad463fdbc6fd2d7d7ab"}
Sep 15 07:19:40.626090 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.626066    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-24824" event={"ID":"af05362c-5502-4642-8611-dca76ad8c1f7","Type":"ContainerStarted","Data":"78ebe04da2fa719285da356e8516099b563a25323cc7c8159c9625bc750b189e"}
Sep 15 07:19:40.627706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.627678    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z" event={"ID":"48b8e7c5-6f1c-4c91-9546-1f8ed041daff","Type":"ContainerStarted","Data":"03c96b534359085723b7dc601c21772ad79f03f0b5ed2afb1a6f858d1f375e53"}
Sep 15 07:19:40.627816 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.627730    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:40.629397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.629368    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rthp" event={"ID":"1d5ee397-08ec-4837-a81f-3e22caddadcc","Type":"ContainerStarted","Data":"569acd8479a3397463dff5ed1a5257a8a2323e56699f00684217d3dd39d200e5"}
Sep 15 07:19:40.629866 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.629837    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:19:40.646289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.646235    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" podStartSLOduration=2.093612405 podStartE2EDuration="7.64621779s" totalImagesPullingTime="5.552605385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:40.644294291 +0000 UTC m=+68.738460562" watchObservedRunningTime="2026-09-15 07:19:40.64621779 +0000 UTC m=+68.740384054"
Sep 15 07:19:40.661163 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:40.660987    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z" podStartSLOduration=2.05042102 podStartE2EDuration="7.660969735s" totalImagesPullingTime="5.610548715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:40.659524081 +0000 UTC m=+68.753690344" watchObservedRunningTime="2026-09-15 07:19:40.660969735 +0000 UTC m=+68.755135990"
Sep 15 07:19:45.646089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:45.646051    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6rthp" event={"ID":"1d5ee397-08ec-4837-a81f-3e22caddadcc","Type":"ContainerStarted","Data":"00c945930cad7e49b46deb470bc2e9525fc8691d8e71e08c63a4f399d8017029"}
Sep 15 07:19:45.647772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:45.647734    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-24824" event={"ID":"af05362c-5502-4642-8611-dca76ad8c1f7","Type":"ContainerStarted","Data":"7f5bf07c0e8761c6cfc6a6342428458005af74ae7fa32700071e1f1420a7798f"}
Sep 15 07:19:45.647902 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:45.647880    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:19:45.663234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:45.663188    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-6rthp" podStartSLOduration=68.709775766 podStartE2EDuration="1m13.66317468s" totalImagesPullingTime="4.953398914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:45.662509261 +0000 UTC m=+73.756675521" watchObservedRunningTime="2026-09-15 07:19:45.66317468 +0000 UTC m=+73.757340943"
Sep 15 07:19:45.677656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:19:45.677611    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-24824" podStartSLOduration=68.739568873 podStartE2EDuration="1m13.677597641s" totalImagesPullingTime="4.938028768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:19:45.677554873 +0000 UTC m=+73.771721152" watchObservedRunningTime="2026-09-15 07:19:45.677597641 +0000 UTC m=+73.771763919"
Sep 15 07:20:09.641610 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:20:09.641485    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:20:09.642068 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:20:09.641639    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:20:09.642068 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:20:09.641732    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:21:13.641712755 +0000 UTC m=+161.735879004 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:20:16.653188 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:20:16.653158    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-24824"
Sep 15 07:20:42.149530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:20:42.149484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:20:42.149960 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:20:42.149627    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:20:42.149960 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:20:42.149700    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:44.149683177 +0000 UTC m=+252.243849416 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : secret "metrics-daemon-secret" not found
Sep 15 07:21:08.735948 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:21:08.735904    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dt7v6" podUID="7ca0b761-ffd4-409c-bcbd-379522595180"
Sep 15 07:21:08.889530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:08.889502    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:21:09.383259 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:21:09.383231    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:21:13.656633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:13.656595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:21:13.657057 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:21:13.656755    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 07:21:13.657057 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:21:13.656833    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls podName:7ca0b761-ffd4-409c-bcbd-379522595180 nodeName:}" failed. No retries permitted until 2026-09-15 07:23:15.656812435 +0000 UTC m=+283.750978681 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls") pod "dns-default-dt7v6" (UID: "7ca0b761-ffd4-409c-bcbd-379522595180") : secret "dns-default-metrics-tls" not found
Sep 15 07:21:20.372857 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:20.372820    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:21:39.976284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:39.976252    2574 generic.go:356] "Generic (PLEG): container finished" podID="48b8e7c5-6f1c-4c91-9546-1f8ed041daff" containerID="03c96b534359085723b7dc601c21772ad79f03f0b5ed2afb1a6f858d1f375e53" exitCode=1
Sep 15 07:21:39.976606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:39.976329    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z" event={"ID":"48b8e7c5-6f1c-4c91-9546-1f8ed041daff","Type":"ContainerDied","Data":"03c96b534359085723b7dc601c21772ad79f03f0b5ed2afb1a6f858d1f375e53"}
Sep 15 07:21:39.976675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:39.976658    2574 scope.go:117] "RemoveContainer" containerID="03c96b534359085723b7dc601c21772ad79f03f0b5ed2afb1a6f858d1f375e53"
Sep 15 07:21:40.628657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:40.628623    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:21:40.981837 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:40.981802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z" event={"ID":"48b8e7c5-6f1c-4c91-9546-1f8ed041daff","Type":"ContainerStarted","Data":"a3be9189ae140493033507714985d853ae9ec5a98ac69f557ad3b0a852d24245"}
Sep 15 07:21:40.982267 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:40.982102    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:21:40.982743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:21:40.982723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c985588b4-46r6z"
Sep 15 07:22:31.426099 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.426066    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gkw9s"]
Sep 15 07:22:31.428065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.428050    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.430009 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.429987    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 07:22:31.430250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.430228    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 07:22:31.430345 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.430325    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 07:22:31.430799 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.430784    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mn7st\""
Sep 15 07:22:31.430884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.430866    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 07:22:31.436693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.436671    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkw9s"]
Sep 15 07:22:31.564597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.564571    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.564699 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.564601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.564699 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.564622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccj8f\" (UniqueName: \"kubernetes.io/projected/da409ae3-c129-48f5-818c-5b132db016f9-kube-api-access-ccj8f\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.564772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.564704    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da409ae3-c129-48f5-818c-5b132db016f9-crio-socket\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.564772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.564726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da409ae3-c129-48f5-818c-5b132db016f9-data-volume\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.665873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.665842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da409ae3-c129-48f5-818c-5b132db016f9-data-volume\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.665957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.665893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.665957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.665913    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.666021 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:31.665994    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:31.666055 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:31.666043    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:31.666087 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:31.666067    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:32.166046917 +0000 UTC m=+240.260213156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:31.666156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.666086    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccj8f\" (UniqueName: \"kubernetes.io/projected/da409ae3-c129-48f5-818c-5b132db016f9-kube-api-access-ccj8f\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.666156 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:31.666111    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:32.166095491 +0000 UTC m=+240.260261732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:31.666233 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.666158    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da409ae3-c129-48f5-818c-5b132db016f9-data-volume\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.666233 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.666196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da409ae3-c129-48f5-818c-5b132db016f9-crio-socket\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.666296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.666245    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da409ae3-c129-48f5-818c-5b132db016f9-crio-socket\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:31.675886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:31.675858    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccj8f\" (UniqueName: \"kubernetes.io/projected/da409ae3-c129-48f5-818c-5b132db016f9-kube-api-access-ccj8f\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:32.169939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:32.169909    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:32.169939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:32.169939    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:32.170159 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:32.170028    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:32.170159 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:32.170087    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:33.170071947 +0000 UTC m=+241.264238187 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:32.170159 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:32.170107    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:32.170328 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:32.170170    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:33.170159364 +0000 UTC m=+241.264325604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:33.170615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:33.170586    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84rbg_912a1a00-fcb7-488e-922c-90863e26694b/dns-node-resolver/0.log"
Sep 15 07:22:33.176894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:33.176873    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:33.176979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:33.176906    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:33.177041 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:33.177004    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:33.177041 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:33.177017    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:33.177141 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:33.177070    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:35.177053829 +0000 UTC m=+243.271220070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:33.177141 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:33.177083    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:35.177077562 +0000 UTC m=+243.271243801 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:34.370258 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:34.370226    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l5q52_16ce4021-b6c1-472c-b4fe-379a5d2bf4a6/node-ca/0.log"
Sep 15 07:22:35.170795 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:35.170765    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vjvf_964bc816-53aa-4e24-bc1c-af2d4dfdf27a/serve-healthcheck-canary/0.log"
Sep 15 07:22:35.191592 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:35.191561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:35.191735 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:35.191597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:35.191735 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:35.191677    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:35.191735 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:35.191714    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:35.191735 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:35.191731    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:39.19171721 +0000 UTC m=+247.285883450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:35.191915 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:35.191767    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:39.191751184 +0000 UTC m=+247.285917454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:39.218298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:39.218263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:39.218298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:39.218298    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:39.218818 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:39.218380    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 07:22:39.218818 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:39.218419    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:39.218818 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:39.218440    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:47.218424211 +0000 UTC m=+255.312590451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : configmap "kube-rbac-proxy" not found
Sep 15 07:22:39.218818 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:39.218459    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:22:47.21844822 +0000 UTC m=+255.312614460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:44.151656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:44.151617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:22:44.152033 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:44.151728    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 07:22:44.152033 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:44.151780    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs podName:531d1557-eab9-45c5-a333-9122556ce383 nodeName:}" failed. No retries permitted until 2026-09-15 07:24:46.15176528 +0000 UTC m=+374.245931519 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs") pod "network-metrics-daemon-kzwq6" (UID: "531d1557-eab9-45c5-a333-9122556ce383") : secret "metrics-daemon-secret" not found
Sep 15 07:22:47.271996 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:47.271958    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:47.271996 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:47.271999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:47.272416 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:47.272105    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:47.272416 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:22:47.272175    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls podName:da409ae3-c129-48f5-818c-5b132db016f9 nodeName:}" failed. No retries permitted until 2026-09-15 07:23:03.272160377 +0000 UTC m=+271.366326630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gkw9s" (UID: "da409ae3-c129-48f5-818c-5b132db016f9") : secret "insights-runtime-extractor-tls" not found
Sep 15 07:22:47.272526 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:47.272505    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da409ae3-c129-48f5-818c-5b132db016f9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:22:54.592165 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.592129    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cd4bbcf87-tkcl4"]
Sep 15 07:22:54.594197 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.594174    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.596187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.596162    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rxml2\""
Sep 15 07:22:54.596291 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.596206    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 07:22:54.596820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.596805    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 07:22:54.596908 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.596848    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 07:22:54.601829 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.601797    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 07:22:54.611679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.611659    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cd4bbcf87-tkcl4"]
Sep 15 07:22:54.722129 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722090    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-image-registry-private-configuration\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407f8dee-2679-4257-a1ea-46f81a37ad93-ca-trust-extracted\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722152    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqkst\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-kube-api-access-cqkst\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722260    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-trusted-ca\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722291    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-installation-pull-secrets\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-certificates\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722352    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-bound-sa-token\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.722540 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.722433    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-tls\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.822943 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.822917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-trusted-ca\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.822948    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-installation-pull-secrets\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.822974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-certificates\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.822997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-bound-sa-token\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-tls\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823236    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-image-registry-private-configuration\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823261    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407f8dee-2679-4257-a1ea-46f81a37ad93-ca-trust-extracted\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823375 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823283    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqkst\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-kube-api-access-cqkst\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823700    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/407f8dee-2679-4257-a1ea-46f81a37ad93-ca-trust-extracted\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.823956 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823926    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-trusted-ca\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.824035 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.823954    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-certificates\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.825629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.825607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-installation-pull-secrets\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.825717 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.825639    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-registry-tls\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.825717 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.825607    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/407f8dee-2679-4257-a1ea-46f81a37ad93-image-registry-private-configuration\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.834050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.834027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-bound-sa-token\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.834140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.834027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqkst\" (UniqueName: \"kubernetes.io/projected/407f8dee-2679-4257-a1ea-46f81a37ad93-kube-api-access-cqkst\") pod \"image-registry-cd4bbcf87-tkcl4\" (UID: \"407f8dee-2679-4257-a1ea-46f81a37ad93\") " pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:54.903888 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:54.903826    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:55.171648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:55.171609    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cd4bbcf87-tkcl4"]
Sep 15 07:22:55.175300 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:22:55.175273    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407f8dee_2679_4257_a1ea_46f81a37ad93.slice/crio-64e6afcec033f05adac49456a3dc56ca0c968af9993175a7071b3304942697ff WatchSource:0}: Error finding container 64e6afcec033f05adac49456a3dc56ca0c968af9993175a7071b3304942697ff: Status 404 returned error can't find the container with id 64e6afcec033f05adac49456a3dc56ca0c968af9993175a7071b3304942697ff
Sep 15 07:22:55.191521 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:55.191497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4" event={"ID":"407f8dee-2679-4257-a1ea-46f81a37ad93","Type":"ContainerStarted","Data":"64e6afcec033f05adac49456a3dc56ca0c968af9993175a7071b3304942697ff"}
Sep 15 07:22:56.196048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:56.196012    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4" event={"ID":"407f8dee-2679-4257-a1ea-46f81a37ad93","Type":"ContainerStarted","Data":"b6c8d95931d1a2eac5ae52f0de8b969de411df6f35e4e4bba4469a42a647231f"}
Sep 15 07:22:56.196451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:56.196245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:22:56.216648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:22:56.216602    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4" podStartSLOduration=2.216587027 podStartE2EDuration="2.216587027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:22:56.216040662 +0000 UTC m=+264.310206925" watchObservedRunningTime="2026-09-15 07:22:56.216587027 +0000 UTC m=+264.310753291"
Sep 15 07:23:03.279318 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:03.279276    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:23:03.281450 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:03.281431    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da409ae3-c129-48f5-818c-5b132db016f9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gkw9s\" (UID: \"da409ae3-c129-48f5-818c-5b132db016f9\") " pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:23:03.539268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:03.539192    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mn7st\""
Sep 15 07:23:03.547961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:03.547941    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gkw9s"
Sep 15 07:23:03.812058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:03.811997    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gkw9s"]
Sep 15 07:23:03.816400 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:23:03.816378    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda409ae3_c129_48f5_818c_5b132db016f9.slice/crio-da0f95f5037b90edd1e68cdd508544268850dd66736d86e020289a8f160fde61 WatchSource:0}: Error finding container da0f95f5037b90edd1e68cdd508544268850dd66736d86e020289a8f160fde61: Status 404 returned error can't find the container with id da0f95f5037b90edd1e68cdd508544268850dd66736d86e020289a8f160fde61
Sep 15 07:23:04.224283 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:04.224253    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkw9s" event={"ID":"da409ae3-c129-48f5-818c-5b132db016f9","Type":"ContainerStarted","Data":"79aebd2e55acb8d91de576f29756abf8963799d4ef1591bf2bc1268d3dbacb0e"}
Sep 15 07:23:04.224416 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:04.224327    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkw9s" event={"ID":"da409ae3-c129-48f5-818c-5b132db016f9","Type":"ContainerStarted","Data":"da0f95f5037b90edd1e68cdd508544268850dd66736d86e020289a8f160fde61"}
Sep 15 07:23:05.229542 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:05.229513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkw9s" event={"ID":"da409ae3-c129-48f5-818c-5b132db016f9","Type":"ContainerStarted","Data":"6c0483982ad7c1522b3020fb3279ff8fd95ac947bd7807502b267247332c9d87"}
Sep 15 07:23:09.243269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:09.243240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gkw9s" event={"ID":"da409ae3-c129-48f5-818c-5b132db016f9","Type":"ContainerStarted","Data":"91b6cb9d59969126c39b30303a3bdcde92a990c76c901c765a25fd0ed179ca23"}
Sep 15 07:23:09.264001 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:09.263957    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gkw9s" podStartSLOduration=33.506043998 podStartE2EDuration="38.263942759s" totalImagesPullingTime="4.757898761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 07:22:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:09.262776941 +0000 UTC m=+277.356943228" watchObservedRunningTime="2026-09-15 07:23:09.263942759 +0000 UTC m=+277.358109021"
Sep 15 07:23:11.890537 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:23:11.890494    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-dt7v6" podUID="7ca0b761-ffd4-409c-bcbd-379522595180"
Sep 15 07:23:12.251206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:12.251177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:14.614525 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.614493    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-287fs"]
Sep 15 07:23:14.616499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.616482    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.620978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.620957    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ch4s\""
Sep 15 07:23:14.621598 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.621578    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 07:23:14.621746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.621627    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 07:23:14.621823 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.621810    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 07:23:14.621919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.621893    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 07:23:14.624209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.624190    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 07:23:14.625468 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.625453    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 07:23:14.652737 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-root\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652740    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652763    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-tls\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652840    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-wtmp\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsc7k\" (UniqueName: \"kubernetes.io/projected/a896d384-9995-4ef8-94c3-ca53ab0864e7-kube-api-access-jsc7k\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-metrics-client-ca\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.652947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652927    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-sys\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.653082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.652947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-textfile\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.653082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.653008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.753855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753831    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-root\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.753938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.753938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-tls\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.753938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753931    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-root\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-wtmp\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.753991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsc7k\" (UniqueName: \"kubernetes.io/projected/a896d384-9995-4ef8-94c3-ca53ab0864e7-kube-api-access-jsc7k\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-metrics-client-ca\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754023    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-sys\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754041    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-textfile\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-sys\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-wtmp\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754464 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754396    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-accelerators-collector-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754464 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-textfile\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.754669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.754649    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a896d384-9995-4ef8-94c3-ca53ab0864e7-metrics-client-ca\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.756333 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.756315    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-tls\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.756417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.756342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a896d384-9995-4ef8-94c3-ca53ab0864e7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.761147 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.761124    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsc7k\" (UniqueName: \"kubernetes.io/projected/a896d384-9995-4ef8-94c3-ca53ab0864e7-kube-api-access-jsc7k\") pod \"node-exporter-287fs\" (UID: \"a896d384-9995-4ef8-94c3-ca53ab0864e7\") " pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:14.908693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.908656    2574 patch_prober.go:28] interesting pod/image-registry-cd4bbcf87-tkcl4 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 07:23:14.908826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.908711    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4" podUID="407f8dee-2679-4257-a1ea-46f81a37ad93" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:23:14.925209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:14.925177    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-287fs"
Sep 15 07:23:15.261353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:15.261291    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287fs" event={"ID":"a896d384-9995-4ef8-94c3-ca53ab0864e7","Type":"ContainerStarted","Data":"4d29b13af500538eabd194860200a7686c1e6e887a602a91ededd2331e53dbf7"}
Sep 15 07:23:15.661432 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:15.661400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:15.663692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:15.663670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7ca0b761-ffd4-409c-bcbd-379522595180-metrics-tls\") pod \"dns-default-dt7v6\" (UID: \"7ca0b761-ffd4-409c-bcbd-379522595180\") " pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:15.854241 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:15.854212    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tfj9f\""
Sep 15 07:23:15.862951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:15.862914    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:16.134022 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:16.133988    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dt7v6"]
Sep 15 07:23:16.170635 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:23:16.170574    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca0b761_ffd4_409c_bcbd_379522595180.slice/crio-d5273c1ae28c7fa3ffd6482dffae5c1fb4d9f6b521b2ea80cab62bd0e830b75c WatchSource:0}: Error finding container d5273c1ae28c7fa3ffd6482dffae5c1fb4d9f6b521b2ea80cab62bd0e830b75c: Status 404 returned error can't find the container with id d5273c1ae28c7fa3ffd6482dffae5c1fb4d9f6b521b2ea80cab62bd0e830b75c
Sep 15 07:23:16.265543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:16.265513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dt7v6" event={"ID":"7ca0b761-ffd4-409c-bcbd-379522595180","Type":"ContainerStarted","Data":"d5273c1ae28c7fa3ffd6482dffae5c1fb4d9f6b521b2ea80cab62bd0e830b75c"}
Sep 15 07:23:17.203821 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:17.203795    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cd4bbcf87-tkcl4"
Sep 15 07:23:17.272868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:17.272824    2574 generic.go:356] "Generic (PLEG): container finished" podID="a896d384-9995-4ef8-94c3-ca53ab0864e7" containerID="041009d277bd3e3468fe4c8adb081fdab727930533c81e879e05ae6d2062433f" exitCode=0
Sep 15 07:23:17.273420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:17.272924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287fs" event={"ID":"a896d384-9995-4ef8-94c3-ca53ab0864e7","Type":"ContainerDied","Data":"041009d277bd3e3468fe4c8adb081fdab727930533c81e879e05ae6d2062433f"}
Sep 15 07:23:18.277819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.277784    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dt7v6" event={"ID":"7ca0b761-ffd4-409c-bcbd-379522595180","Type":"ContainerStarted","Data":"33498285a5b43b1f92a95a55db3cc91c45e7b486c4a86c4c0508cc4585f1c1ac"}
Sep 15 07:23:18.278245 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.277872    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dt7v6" event={"ID":"7ca0b761-ffd4-409c-bcbd-379522595180","Type":"ContainerStarted","Data":"4ef78c912bda68070d213d6e1266537e7001d6f25007470914fd0cfd61259cda"}
Sep 15 07:23:18.278245 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.277901    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:18.280011 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.279990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287fs" event={"ID":"a896d384-9995-4ef8-94c3-ca53ab0864e7","Type":"ContainerStarted","Data":"7d92a099166453fd2f375a1681c48007275ad9c3cd3a95f9f70defdd070fc26d"}
Sep 15 07:23:18.280108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.280034    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-287fs" event={"ID":"a896d384-9995-4ef8-94c3-ca53ab0864e7","Type":"ContainerStarted","Data":"304274c04b7db7ebfb1897e014c7664915afb8d9a800f386d1cef1a7b0d9e8ab"}
Sep 15 07:23:18.295189 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.295153    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dt7v6" podStartSLOduration=251.820762887 podStartE2EDuration="4m13.295140791s" totalImagesPullingTime="1.474377904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:18.294109437 +0000 UTC m=+286.388275731" watchObservedRunningTime="2026-09-15 07:23:18.295140791 +0000 UTC m=+286.389307071"
Sep 15 07:23:18.312917 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:18.312876    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-287fs" podStartSLOduration=2.99112386 podStartE2EDuration="4.312864979s" totalImagesPullingTime="1.262479601s" totalInitContainerRuntime="59.261518ms" isStatefulPod=true podCreationTimestamp="2026-09-15 07:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:18.311791451 +0000 UTC m=+286.405957754" watchObservedRunningTime="2026-09-15 07:23:18.312864979 +0000 UTC m=+286.407031240"
Sep 15 07:23:23.373566 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:23:23.373528    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-kzwq6" podUID="531d1557-eab9-45c5-a333-9122556ce383"
Sep 15 07:23:23.916882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:23.916840    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" podUID="2aded570-6864-4ffa-abc1-f6d529e8c81b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 07:23:28.286934 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:28.286898    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dt7v6"
Sep 15 07:23:32.257138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:32.257096    2574 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 07:23:33.917157 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:33.917091    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" podUID="2aded570-6864-4ffa-abc1-f6d529e8c81b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 07:23:36.372537 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:36.372504    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:23:41.752393 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.752352    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:23:41.758237 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.758212    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.760762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.760742    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 07:23:41.761328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761308    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 07:23:41.761409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761356    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4rwg4\""
Sep 15 07:23:41.761409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761364    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 07:23:41.761515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761407    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 07:23:41.761515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761422    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 07:23:41.761515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761497    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 07:23:41.761690 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.761676    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 07:23:41.765560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.765537    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:23:41.766929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.766907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 07:23:41.836984 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.836962    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837075 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.836986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837075 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.837003    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837075 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.837034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.837102    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.837151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.837222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.837178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxvm\" (UniqueName: \"kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938198 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxvm\" (UniqueName: \"kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938251    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938264    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.938905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938882    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.939010 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.939010 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.938973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.939272 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.939255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.941355 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.941328    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.941433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.941336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:41.945420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:41.945399    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxvm\" (UniqueName: \"kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm\") pod \"console-7f8b6478f7-zn4jc\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") " pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:42.069495 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:42.069439    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:42.341276 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:42.341207    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:23:42.343842 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:23:42.343814    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5d34c96_914a_4c69_b3a7_dfe963feccb9.slice/crio-9265c9cc6de19c7f4b11a9098fc4906376ac9386532069b0aa09b75c13850c7a WatchSource:0}: Error finding container 9265c9cc6de19c7f4b11a9098fc4906376ac9386532069b0aa09b75c13850c7a: Status 404 returned error can't find the container with id 9265c9cc6de19c7f4b11a9098fc4906376ac9386532069b0aa09b75c13850c7a
Sep 15 07:23:42.345878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:42.345862    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:23:42.364896 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:42.364873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b6478f7-zn4jc" event={"ID":"e5d34c96-914a-4c69-b3a7-dfe963feccb9","Type":"ContainerStarted","Data":"9265c9cc6de19c7f4b11a9098fc4906376ac9386532069b0aa09b75c13850c7a"}
Sep 15 07:23:43.916710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:43.916661    2574 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" podUID="2aded570-6864-4ffa-abc1-f6d529e8c81b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 07:23:43.917201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:43.916743    2574 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb"
Sep 15 07:23:43.917399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:43.917361    2574 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"1e95f99522d600ce98917653d239c05e896cf00e0652d77936637db89b566b64"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 15 07:23:43.917441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:43.917428    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" podUID="2aded570-6864-4ffa-abc1-f6d529e8c81b" containerName="service-proxy" containerID="cri-o://1e95f99522d600ce98917653d239c05e896cf00e0652d77936637db89b566b64" gracePeriod=30
Sep 15 07:23:44.374024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:44.373931    2574 generic.go:356] "Generic (PLEG): container finished" podID="2aded570-6864-4ffa-abc1-f6d529e8c81b" containerID="1e95f99522d600ce98917653d239c05e896cf00e0652d77936637db89b566b64" exitCode=2
Sep 15 07:23:44.377968 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:44.377935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerDied","Data":"1e95f99522d600ce98917653d239c05e896cf00e0652d77936637db89b566b64"}
Sep 15 07:23:44.378095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:44.377984    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-675bcfb866-kt4mb" event={"ID":"2aded570-6864-4ffa-abc1-f6d529e8c81b","Type":"ContainerStarted","Data":"b89e6b69bf5343c91c705c1ec866fbae8ba2df130e166ff6bfc218a81d0dc791"}
Sep 15 07:23:45.378525 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:45.378489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b6478f7-zn4jc" event={"ID":"e5d34c96-914a-4c69-b3a7-dfe963feccb9","Type":"ContainerStarted","Data":"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"}
Sep 15 07:23:45.396373 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:45.396289    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f8b6478f7-zn4jc" podStartSLOduration=1.6048352320000001 podStartE2EDuration="4.396274326s" totalImagesPullingTime="2.791439094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:23:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:23:45.395596765 +0000 UTC m=+313.489763039" watchObservedRunningTime="2026-09-15 07:23:45.396274326 +0000 UTC m=+313.490440642"
Sep 15 07:23:52.070043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:52.070007    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:52.070520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:52.070080    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:52.075440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:52.075413    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:23:52.405004 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:23:52.404924    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:24:46.175745 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.175706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:24:46.178176 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.178151    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/531d1557-eab9-45c5-a333-9122556ce383-metrics-certs\") pod \"network-metrics-daemon-kzwq6\" (UID: \"531d1557-eab9-45c5-a333-9122556ce383\") " pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:24:46.274885 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.274856    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ftj64\""
Sep 15 07:24:46.283392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.283366    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kzwq6"
Sep 15 07:24:46.561647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.561578    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kzwq6"]
Sep 15 07:24:46.564388 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:24:46.564362    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod531d1557_eab9_45c5_a333_9122556ce383.slice/crio-f5d51b308f8a42fdf2f474ca473ef2cc08ca33e27903c38d01e0c23e7633bad0 WatchSource:0}: Error finding container f5d51b308f8a42fdf2f474ca473ef2cc08ca33e27903c38d01e0c23e7633bad0: Status 404 returned error can't find the container with id f5d51b308f8a42fdf2f474ca473ef2cc08ca33e27903c38d01e0c23e7633bad0
Sep 15 07:24:46.570409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:46.570378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzwq6" event={"ID":"531d1557-eab9-45c5-a333-9122556ce383","Type":"ContainerStarted","Data":"f5d51b308f8a42fdf2f474ca473ef2cc08ca33e27903c38d01e0c23e7633bad0"}
Sep 15 07:24:48.578503 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:48.578468    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzwq6" event={"ID":"531d1557-eab9-45c5-a333-9122556ce383","Type":"ContainerStarted","Data":"c606831ffe182fb97a12d88332d4085c05a86477ffe5fd02fb7a57fde9d0235a"}
Sep 15 07:24:48.578989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:48.578536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kzwq6" event={"ID":"531d1557-eab9-45c5-a333-9122556ce383","Type":"ContainerStarted","Data":"03d59f9b1e7319d2ed7d4c3e380022bd358431e713982cb813d33a8a58ecdbe9"}
Sep 15 07:24:48.594625 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:48.594582    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kzwq6" podStartSLOduration=375.626024593 podStartE2EDuration="6m16.594567942s" totalImagesPullingTime="968.543349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:24:48.593843052 +0000 UTC m=+376.688009311" watchObservedRunningTime="2026-09-15 07:24:48.594567942 +0000 UTC m=+376.688734204"
Sep 15 07:24:52.797901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:24:52.797866    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:25:17.817629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:17.817564    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7f8b6478f7-zn4jc" podUID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" containerName="console" containerID="cri-o://83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a" gracePeriod=15
Sep 15 07:25:18.051450 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.051427    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:25:18.088314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088285    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088316    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088335    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088366    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgxvm\" (UniqueName: \"kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088390    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088422    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config\") pod \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\" (UID: \"e5d34c96-914a-4c69-b3a7-dfe963feccb9\") "
Sep 15 07:25:18.088777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088746    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config" (OuterVolumeSpecName: "console-config") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:25:18.088910 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088771    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:25:18.088978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.088932    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca" (OuterVolumeSpecName: "service-ca") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:25:18.089039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.089003    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:25:18.090644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.090621    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm" (OuterVolumeSpecName: "kube-api-access-xgxvm") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "kube-api-access-xgxvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:25:18.090744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.090617    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:25:18.090744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.090629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e5d34c96-914a-4c69-b3a7-dfe963feccb9" (UID: "e5d34c96-914a-4c69-b3a7-dfe963feccb9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:25:18.189262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189231    2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-trusted-ca-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189258    2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-oauth-serving-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189268    2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-config\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189276    2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e5d34c96-914a-4c69-b3a7-dfe963feccb9-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189284    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgxvm\" (UniqueName: \"kubernetes.io/projected/e5d34c96-914a-4c69-b3a7-dfe963feccb9-kube-api-access-xgxvm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189298    2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-serving-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.189409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.189306    2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e5d34c96-914a-4c69-b3a7-dfe963feccb9-console-oauth-config\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:25:18.678852 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.678819    2574 generic.go:356] "Generic (PLEG): container finished" podID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" containerID="83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a" exitCode=0
Sep 15 07:25:18.679026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.678873    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f8b6478f7-zn4jc"
Sep 15 07:25:18.679026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.678917    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b6478f7-zn4jc" event={"ID":"e5d34c96-914a-4c69-b3a7-dfe963feccb9","Type":"ContainerDied","Data":"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"}
Sep 15 07:25:18.679026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.679000    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f8b6478f7-zn4jc" event={"ID":"e5d34c96-914a-4c69-b3a7-dfe963feccb9","Type":"ContainerDied","Data":"9265c9cc6de19c7f4b11a9098fc4906376ac9386532069b0aa09b75c13850c7a"}
Sep 15 07:25:18.679222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.679028    2574 scope.go:117] "RemoveContainer" containerID="83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"
Sep 15 07:25:18.687969 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.687954    2574 scope.go:117] "RemoveContainer" containerID="83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"
Sep 15 07:25:18.688337 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:25:18.688317    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a\": container with ID starting with 83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a not found: ID does not exist" containerID="83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"
Sep 15 07:25:18.688399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.688345    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a"} err="failed to get container status \"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a\": rpc error: code = NotFound desc = could not find container \"83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a\": container with ID starting with 83fa227ad868f430aeefc983d7b675e1c0bb634ffc055ad25bc1980c163d5a5a not found: ID does not exist"
Sep 15 07:25:18.693481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.693458    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:25:18.696826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:18.696806    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7f8b6478f7-zn4jc"]
Sep 15 07:25:20.376788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:25:20.376747    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" path="/var/lib/kubelet/pods/e5d34c96-914a-4c69-b3a7-dfe963feccb9/volumes"
Sep 15 07:26:38.198288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.198243    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"]
Sep 15 07:26:38.198811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.198506    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" containerName="console"
Sep 15 07:26:38.198811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.198519    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" containerName="console"
Sep 15 07:26:38.198811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.198597    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5d34c96-914a-4c69-b3a7-dfe963feccb9" containerName="console"
Sep 15 07:26:38.200782 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.200750    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.203946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.203920    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 07:26:38.204069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.204011    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 07:26:38.204890 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.204868    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nftv2\""
Sep 15 07:26:38.216227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.216197    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"]
Sep 15 07:26:38.301043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.301023    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjtd\" (UniqueName: \"kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.301180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.301077    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.301180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.301144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.402162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.402141    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjtd\" (UniqueName: \"kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.402251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.402174    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.402251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.402191    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.402501 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.402484    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.402582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.402565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.411415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.411387    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjtd\" (UniqueName: \"kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.510439 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.510389    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:26:38.777786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.777655    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"]
Sep 15 07:26:38.780286 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:26:38.780256    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6935ede0_54f4_41fb_a728_009a247ce633.slice/crio-d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5 WatchSource:0}: Error finding container d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5: Status 404 returned error can't find the container with id d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5
Sep 15 07:26:38.951760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:38.951732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz" event={"ID":"6935ede0-54f4-41fb-a728-009a247ce633","Type":"ContainerStarted","Data":"d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5"}
Sep 15 07:26:47.982588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:47.982560    2574 generic.go:356] "Generic (PLEG): container finished" podID="6935ede0-54f4-41fb-a728-009a247ce633" containerID="25c589326e055267dbc65ffe4964ee1a80cc3ec5bb76c579f134bee7bef5347c" exitCode=0
Sep 15 07:26:47.982912 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:47.982608    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz" event={"ID":"6935ede0-54f4-41fb-a728-009a247ce633","Type":"ContainerDied","Data":"25c589326e055267dbc65ffe4964ee1a80cc3ec5bb76c579f134bee7bef5347c"}
Sep 15 07:26:50.995020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:50.994978    2574 generic.go:356] "Generic (PLEG): container finished" podID="6935ede0-54f4-41fb-a728-009a247ce633" containerID="a686f58f8f1212d9e9f24f844b3a71a289d1f9cc85628039ee834219ed77ddbd" exitCode=0
Sep 15 07:26:50.995020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:50.995024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz" event={"ID":"6935ede0-54f4-41fb-a728-009a247ce633","Type":"ContainerDied","Data":"a686f58f8f1212d9e9f24f844b3a71a289d1f9cc85628039ee834219ed77ddbd"}
Sep 15 07:26:59.025177 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:59.025138    2574 generic.go:356] "Generic (PLEG): container finished" podID="6935ede0-54f4-41fb-a728-009a247ce633" containerID="fe08bfcac9315b88785c25fc8f918a5828bc3271f26324261f0e942ac1d2df58" exitCode=0
Sep 15 07:26:59.025533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:26:59.025211    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz" event={"ID":"6935ede0-54f4-41fb-a728-009a247ce633","Type":"ContainerDied","Data":"fe08bfcac9315b88785c25fc8f918a5828bc3271f26324261f0e942ac1d2df58"}
Sep 15 07:27:00.156740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.156718    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:27:00.250496 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.250454    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jjtd\" (UniqueName: \"kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd\") pod \"6935ede0-54f4-41fb-a728-009a247ce633\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") "
Sep 15 07:27:00.250681 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.250531    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util\") pod \"6935ede0-54f4-41fb-a728-009a247ce633\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") "
Sep 15 07:27:00.250681 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.250555    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle\") pod \"6935ede0-54f4-41fb-a728-009a247ce633\" (UID: \"6935ede0-54f4-41fb-a728-009a247ce633\") "
Sep 15 07:27:00.251180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.251147    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle" (OuterVolumeSpecName: "bundle") pod "6935ede0-54f4-41fb-a728-009a247ce633" (UID: "6935ede0-54f4-41fb-a728-009a247ce633"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:00.252659 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.252633    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd" (OuterVolumeSpecName: "kube-api-access-2jjtd") pod "6935ede0-54f4-41fb-a728-009a247ce633" (UID: "6935ede0-54f4-41fb-a728-009a247ce633"). InnerVolumeSpecName "kube-api-access-2jjtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:27:00.254599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.254581    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util" (OuterVolumeSpecName: "util") pod "6935ede0-54f4-41fb-a728-009a247ce633" (UID: "6935ede0-54f4-41fb-a728-009a247ce633"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:00.351282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.351224    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:00.351282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.351247    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6935ede0-54f4-41fb-a728-009a247ce633-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:00.351282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:00.351257    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjtd\" (UniqueName: \"kubernetes.io/projected/6935ede0-54f4-41fb-a728-009a247ce633-kube-api-access-2jjtd\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:01.033559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:01.033528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz" event={"ID":"6935ede0-54f4-41fb-a728-009a247ce633","Type":"ContainerDied","Data":"d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5"}
Sep 15 07:27:01.033715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:01.033568    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540s74dz"
Sep 15 07:27:01.033715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:01.033572    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1c5b025ff67def283ede546f0e796fd23752e4b7d19b17785330f18fe4084d5"
Sep 15 07:27:06.254396 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254361    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"]
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254592    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="util"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254603    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="util"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254610    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="pull"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254616    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="pull"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254627    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="extract"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254633    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="extract"
Sep 15 07:27:06.254832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.254679    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6935ede0-54f4-41fb-a728-009a247ce633" containerName="extract"
Sep 15 07:27:06.258178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.258163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.260979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.260954    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 07:27:06.260979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.260956    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:27:06.261202 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.261000    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-252rl\""
Sep 15 07:27:06.270640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.270619    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"]
Sep 15 07:27:06.388997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.388972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h56ql\" (UniqueName: \"kubernetes.io/projected/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-kube-api-access-h56ql\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.389129 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.389001    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.489860 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.489833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h56ql\" (UniqueName: \"kubernetes.io/projected/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-kube-api-access-h56ql\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.489989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.489864    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.490205 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.490187    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.497585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.497564    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h56ql\" (UniqueName: \"kubernetes.io/projected/7d1ac2eb-a8bb-466d-95df-95d55f11ad4a-kube-api-access-h56ql\") pod \"cert-manager-operator-controller-manager-55c444bf86-j9c88\" (UID: \"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.567092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.567028    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"
Sep 15 07:27:06.845081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:06.845057    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88"]
Sep 15 07:27:06.848001 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:06.847968    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1ac2eb_a8bb_466d_95df_95d55f11ad4a.slice/crio-5919e76a86b848e6a4752383cf14a786a3a57c5910ab688655f02a09d7742bb2 WatchSource:0}: Error finding container 5919e76a86b848e6a4752383cf14a786a3a57c5910ab688655f02a09d7742bb2: Status 404 returned error can't find the container with id 5919e76a86b848e6a4752383cf14a786a3a57c5910ab688655f02a09d7742bb2
Sep 15 07:27:07.054248 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:07.054192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88" event={"ID":"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a","Type":"ContainerStarted","Data":"5919e76a86b848e6a4752383cf14a786a3a57c5910ab688655f02a09d7742bb2"}
Sep 15 07:27:09.064273 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:09.064233    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88" event={"ID":"7d1ac2eb-a8bb-466d-95df-95d55f11ad4a","Type":"ContainerStarted","Data":"9e293944206aac0cfddc0d693bfa9e8038ae414ecfa413ca81fc988a5b1fc036"}
Sep 15 07:27:09.085018 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:09.084961    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-j9c88" podStartSLOduration=1.6439109840000001 podStartE2EDuration="3.084942614s" totalImagesPullingTime="1.44103163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:09.083518348 +0000 UTC m=+517.177684622" watchObservedRunningTime="2026-09-15 07:27:09.084942614 +0000 UTC m=+517.179108877"
Sep 15 07:27:10.169855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.169816    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"]
Sep 15 07:27:10.173972 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.173945    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.176797 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.176775    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 07:27:10.177361 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.177340    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nftv2\""
Sep 15 07:27:10.177361 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.177354    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 07:27:10.186243 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.186221    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"]
Sep 15 07:27:10.320318 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.320279    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8754v\" (UniqueName: \"kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.320488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.320331    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.320488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.320407    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.420800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.420728    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.420800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.420776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.420965 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.420811    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8754v\" (UniqueName: \"kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.421210 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.421191    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.421261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.421206    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.428664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.428645    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8754v\" (UniqueName: \"kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.482517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.482492    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:10.751371 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:10.751302    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"]
Sep 15 07:27:10.754221 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:10.754194    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619295e3_456e_4601_b2ae_8e3a79d3358d.slice/crio-04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40 WatchSource:0}: Error finding container 04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40: Status 404 returned error can't find the container with id 04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40
Sep 15 07:27:11.072605 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:11.072528    2574 generic.go:356] "Generic (PLEG): container finished" podID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerID="34133c3084b68ffaa84a35367a2eaf6c844d83a60f63d00d642c2958f1e17ad2" exitCode=0
Sep 15 07:27:11.072746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:11.072610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw" event={"ID":"619295e3-456e-4601-b2ae-8e3a79d3358d","Type":"ContainerDied","Data":"34133c3084b68ffaa84a35367a2eaf6c844d83a60f63d00d642c2958f1e17ad2"}
Sep 15 07:27:11.072746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:11.072661    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw" event={"ID":"619295e3-456e-4601-b2ae-8e3a79d3358d","Type":"ContainerStarted","Data":"04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40"}
Sep 15 07:27:14.085081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:14.085046    2574 generic.go:356] "Generic (PLEG): container finished" podID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerID="a2ea7105633fdd08908bd3ac08e987f4e5b475fb60e0b23914075048b467a3e9" exitCode=0
Sep 15 07:27:14.085081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:14.085083    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw" event={"ID":"619295e3-456e-4601-b2ae-8e3a79d3358d","Type":"ContainerDied","Data":"a2ea7105633fdd08908bd3ac08e987f4e5b475fb60e0b23914075048b467a3e9"}
Sep 15 07:27:15.090464 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:15.090427    2574 generic.go:356] "Generic (PLEG): container finished" podID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerID="ee55ebfa2f49b61677d70f1410a23f639439fad4191fefde398e0dfe526ba0a9" exitCode=0
Sep 15 07:27:15.090821 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:15.090471    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw" event={"ID":"619295e3-456e-4601-b2ae-8e3a79d3358d","Type":"ContainerDied","Data":"ee55ebfa2f49b61677d70f1410a23f639439fad4191fefde398e0dfe526ba0a9"}
Sep 15 07:27:16.221032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.221007    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:16.255076 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.255049    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util\") pod \"619295e3-456e-4601-b2ae-8e3a79d3358d\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") "
Sep 15 07:27:16.255233 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.255087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle\") pod \"619295e3-456e-4601-b2ae-8e3a79d3358d\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") "
Sep 15 07:27:16.255233 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.255110    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8754v\" (UniqueName: \"kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v\") pod \"619295e3-456e-4601-b2ae-8e3a79d3358d\" (UID: \"619295e3-456e-4601-b2ae-8e3a79d3358d\") "
Sep 15 07:27:16.255530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.255500    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle" (OuterVolumeSpecName: "bundle") pod "619295e3-456e-4601-b2ae-8e3a79d3358d" (UID: "619295e3-456e-4601-b2ae-8e3a79d3358d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:16.257157 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.257132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v" (OuterVolumeSpecName: "kube-api-access-8754v") pod "619295e3-456e-4601-b2ae-8e3a79d3358d" (UID: "619295e3-456e-4601-b2ae-8e3a79d3358d"). InnerVolumeSpecName "kube-api-access-8754v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:27:16.356334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.356277    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:16.356334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.356301    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8754v\" (UniqueName: \"kubernetes.io/projected/619295e3-456e-4601-b2ae-8e3a79d3358d-kube-api-access-8754v\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:16.435989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.435954    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util" (OuterVolumeSpecName: "util") pod "619295e3-456e-4601-b2ae-8e3a79d3358d" (UID: "619295e3-456e-4601-b2ae-8e3a79d3358d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:16.457161 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:16.457138    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/619295e3-456e-4601-b2ae-8e3a79d3358d-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:17.099723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:17.099685    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw" event={"ID":"619295e3-456e-4601-b2ae-8e3a79d3358d","Type":"ContainerDied","Data":"04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40"}
Sep 15 07:27:17.099723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:17.099705    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fptfkw"
Sep 15 07:27:17.099723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:17.099732    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04b115069ac58e2cb85fb3141e30ab7cf373ffd69c2af94cf6d926c3ced85a40"
Sep 15 07:27:22.897648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897615    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"]
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897840    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="extract"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897850    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="extract"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897864    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="util"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897870    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="util"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897876    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="pull"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897882    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="pull"
Sep 15 07:27:22.898085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.897958    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="619295e3-456e-4601-b2ae-8e3a79d3358d" containerName="extract"
Sep 15 07:27:22.899724 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.899707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:22.902072 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.902053    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 07:27:22.902201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.902068    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 07:27:22.902761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.902746    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-78pft\""
Sep 15 07:27:22.908088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.908067    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"]
Sep 15 07:27:22.994861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.994832    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qsvd\" (UniqueName: \"kubernetes.io/projected/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-kube-api-access-7qsvd\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:22.994981 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:22.994875    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.095469 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.095441    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qsvd\" (UniqueName: \"kubernetes.io/projected/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-kube-api-access-7qsvd\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.095593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.095484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.095882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.095864    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-tmp\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.103709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.103686    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qsvd\" (UniqueName: \"kubernetes.io/projected/347dc5f0-3296-4fc9-85bb-906a75d4e6cf-kube-api-access-7qsvd\") pod \"openshift-lws-operator-fd8ccff4c-lknc8\" (UID: \"347dc5f0-3296-4fc9-85bb-906a75d4e6cf\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.208793 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.208771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"
Sep 15 07:27:23.473778 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:23.473707    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8"]
Sep 15 07:27:23.476651 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:23.476622    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod347dc5f0_3296_4fc9_85bb_906a75d4e6cf.slice/crio-54a20105cdf541bf3d34610f1f89971cb2b0f528a36d49e42c2c88817b1c2fb0 WatchSource:0}: Error finding container 54a20105cdf541bf3d34610f1f89971cb2b0f528a36d49e42c2c88817b1c2fb0: Status 404 returned error can't find the container with id 54a20105cdf541bf3d34610f1f89971cb2b0f528a36d49e42c2c88817b1c2fb0
Sep 15 07:27:24.124225 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:24.124189    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8" event={"ID":"347dc5f0-3296-4fc9-85bb-906a75d4e6cf","Type":"ContainerStarted","Data":"54a20105cdf541bf3d34610f1f89971cb2b0f528a36d49e42c2c88817b1c2fb0"}
Sep 15 07:27:26.133891 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:26.133809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8" event={"ID":"347dc5f0-3296-4fc9-85bb-906a75d4e6cf","Type":"ContainerStarted","Data":"6e3e796e85f0354a4b7af204ac2fa5b7679bab2382b627aa726946e2d45325e3"}
Sep 15 07:27:26.148560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:26.148514    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-lknc8" podStartSLOduration=1.928747017 podStartE2EDuration="4.148500125s" totalImagesPullingTime="2.219753108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:26.147141449 +0000 UTC m=+534.241307709" watchObservedRunningTime="2026-09-15 07:27:26.148500125 +0000 UTC m=+534.242666387"
Sep 15 07:27:28.271383 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.271347    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"]
Sep 15 07:27:28.290452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.290422    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"]
Sep 15 07:27:28.290598 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.290511    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.293608 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.293586    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nftv2\""
Sep 15 07:27:28.293794 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.293780    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 07:27:28.295026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.295013    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 07:27:28.332565 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.332541    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.332700 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.332580    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn56f\" (UniqueName: \"kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.332700 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.332680    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.433431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.433404    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.433541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.433449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.433541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.433468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn56f\" (UniqueName: \"kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.433849 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.433826    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.433887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.433837    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.441510 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.441484    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn56f\" (UniqueName: \"kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.599294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.599237    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:28.885535 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:28.885456    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"]
Sep 15 07:27:28.888931 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:28.888898    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98ef030_0e7c_4ede_b763_df5ab0a41bdf.slice/crio-f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c WatchSource:0}: Error finding container f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c: Status 404 returned error can't find the container with id f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c
Sep 15 07:27:29.146964 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:29.146895    2574 generic.go:356] "Generic (PLEG): container finished" podID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerID="5ef64ce396bc27441abf0df50f9cce21ff4d78e159501f1b81c681e203d40709" exitCode=0
Sep 15 07:27:29.146964 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:29.146935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x" event={"ID":"e98ef030-0e7c-4ede-b763-df5ab0a41bdf","Type":"ContainerDied","Data":"5ef64ce396bc27441abf0df50f9cce21ff4d78e159501f1b81c681e203d40709"}
Sep 15 07:27:29.147137 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:29.146968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x" event={"ID":"e98ef030-0e7c-4ede-b763-df5ab0a41bdf","Type":"ContainerStarted","Data":"f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c"}
Sep 15 07:27:30.153849 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:30.153812    2574 generic.go:356] "Generic (PLEG): container finished" podID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerID="47f70ae7b591e8f9609fb71c6398a04de1ae40028dd746c192fd0258092a4d3b" exitCode=0
Sep 15 07:27:30.154317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:30.153855    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x" event={"ID":"e98ef030-0e7c-4ede-b763-df5ab0a41bdf","Type":"ContainerDied","Data":"47f70ae7b591e8f9609fb71c6398a04de1ae40028dd746c192fd0258092a4d3b"}
Sep 15 07:27:31.160635 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:31.160604    2574 generic.go:356] "Generic (PLEG): container finished" podID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerID="016f27f4daf97b2d6bbca42c7a5b43e43dfab3ce467d665f531f3fc818cd792c" exitCode=0
Sep 15 07:27:31.161001 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:31.160657    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x" event={"ID":"e98ef030-0e7c-4ede-b763-df5ab0a41bdf","Type":"ContainerDied","Data":"016f27f4daf97b2d6bbca42c7a5b43e43dfab3ce467d665f531f3fc818cd792c"}
Sep 15 07:27:32.290009 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.289986    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:32.361601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.361579    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle\") pod \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") "
Sep 15 07:27:32.361733 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.361625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util\") pod \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") "
Sep 15 07:27:32.361733 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.361660    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn56f\" (UniqueName: \"kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f\") pod \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\" (UID: \"e98ef030-0e7c-4ede-b763-df5ab0a41bdf\") "
Sep 15 07:27:32.362294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.362268    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle" (OuterVolumeSpecName: "bundle") pod "e98ef030-0e7c-4ede-b763-df5ab0a41bdf" (UID: "e98ef030-0e7c-4ede-b763-df5ab0a41bdf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:32.363619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.363600    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f" (OuterVolumeSpecName: "kube-api-access-rn56f") pod "e98ef030-0e7c-4ede-b763-df5ab0a41bdf" (UID: "e98ef030-0e7c-4ede-b763-df5ab0a41bdf"). InnerVolumeSpecName "kube-api-access-rn56f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:27:32.369723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.369689    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util" (OuterVolumeSpecName: "util") pod "e98ef030-0e7c-4ede-b763-df5ab0a41bdf" (UID: "e98ef030-0e7c-4ede-b763-df5ab0a41bdf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:32.462242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.462174    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn56f\" (UniqueName: \"kubernetes.io/projected/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-kube-api-access-rn56f\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:32.462242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.462198    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:32.462242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:32.462207    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e98ef030-0e7c-4ede-b763-df5ab0a41bdf-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:33.169892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:33.169856    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x" event={"ID":"e98ef030-0e7c-4ede-b763-df5ab0a41bdf","Type":"ContainerDied","Data":"f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c"}
Sep 15 07:27:33.170138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:33.169906    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8a80af0624c3733892e7a4b7f9257fa327b7430d339c4d18242d85bf1324b5c"
Sep 15 07:27:33.170138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:33.169872    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afp585x"
Sep 15 07:27:46.326533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326501    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"]
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326746    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="pull"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326756    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="pull"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326764    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="extract"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326769    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="extract"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326780    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="util"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326785    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="util"
Sep 15 07:27:46.326925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.326828    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e98ef030-0e7c-4ede-b763-df5ab0a41bdf" containerName="extract"
Sep 15 07:27:46.329431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.329413    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.331601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.331572    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 07:27:46.331703 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.331623    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 07:27:46.331865 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.331851    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 07:27:46.331948 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.331878    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qd2gg\""
Sep 15 07:27:46.331948 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.331884    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 07:27:46.345649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.345623    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"]
Sep 15 07:27:46.353506 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.353483    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.353601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.353517    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-kube-api-access-9x4q2\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.353601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.353553    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.353601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.353594    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.353716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.353613    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.454810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.454781    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.454983 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.454818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.454983 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.454840    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-kube-api-access-9x4q2\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.454983 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.454883    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.455174 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.454991    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.455240 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.455203    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-manifests\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.457209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.457176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-tmp\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.457326 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.457266    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.457391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.457374    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.475576 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.475558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x4q2\" (UniqueName: \"kubernetes.io/projected/a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520-kube-api-access-9x4q2\") pod \"opendatahub-operator-controller-manager-565d545f7b-tsjj8\" (UID: \"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520\") " pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.639363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.639284    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:46.888883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.888850    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"]
Sep 15 07:27:46.891259 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.891208    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:46.893411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.893387    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 07:27:46.893504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.893414    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nftv2\""
Sep 15 07:27:46.893504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.893387    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 07:27:46.899696 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.899672    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"]
Sep 15 07:27:46.912026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.911968    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"]
Sep 15 07:27:46.916319 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:46.916298    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b9a9d3_0bb7_4f86_ac16_8ca61b69c520.slice/crio-fb4695a8c3bf57504f32def09e2cf1034400013453101ab998c7701f3d60d8b9 WatchSource:0}: Error finding container fb4695a8c3bf57504f32def09e2cf1034400013453101ab998c7701f3d60d8b9: Status 404 returned error can't find the container with id fb4695a8c3bf57504f32def09e2cf1034400013453101ab998c7701f3d60d8b9
Sep 15 07:27:46.958721 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.958692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbq2\" (UniqueName: \"kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:46.958843 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.958739    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:46.958843 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:46.958767    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.059441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.059407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbq2\" (UniqueName: \"kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.059441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.059442    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.059642 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.059463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.059802 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.059787    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.059857 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.059839    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.086237 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.086209    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbq2\" (UniqueName: \"kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.201962 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.201934    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:47.224066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.224041    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8" event={"ID":"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520","Type":"ContainerStarted","Data":"fb4695a8c3bf57504f32def09e2cf1034400013453101ab998c7701f3d60d8b9"}
Sep 15 07:27:47.469737 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:47.469674    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"]
Sep 15 07:27:47.473206 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:27:47.473178    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7544daa4_e98e_4d05_9040_007114388553.slice/crio-30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230 WatchSource:0}: Error finding container 30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230: Status 404 returned error can't find the container with id 30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230
Sep 15 07:27:48.230670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:48.230636    2574 generic.go:356] "Generic (PLEG): container finished" podID="7544daa4-e98e-4d05-9040-007114388553" containerID="bf213639476cc6f6a294d82885ee52d7896bd143ae58fbc1bf0d7d2d21928def" exitCode=0
Sep 15 07:27:48.230830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:48.230703    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn" event={"ID":"7544daa4-e98e-4d05-9040-007114388553","Type":"ContainerDied","Data":"bf213639476cc6f6a294d82885ee52d7896bd143ae58fbc1bf0d7d2d21928def"}
Sep 15 07:27:48.230830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:48.230738    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn" event={"ID":"7544daa4-e98e-4d05-9040-007114388553","Type":"ContainerStarted","Data":"30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230"}
Sep 15 07:27:49.236043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:49.236014    2574 generic.go:356] "Generic (PLEG): container finished" podID="7544daa4-e98e-4d05-9040-007114388553" containerID="1b3ba56299bd17a5c96eefaf42f410619d759987f74129c3ff71d7e7a6b517ce" exitCode=0
Sep 15 07:27:49.236469 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:49.236048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn" event={"ID":"7544daa4-e98e-4d05-9040-007114388553","Type":"ContainerDied","Data":"1b3ba56299bd17a5c96eefaf42f410619d759987f74129c3ff71d7e7a6b517ce"}
Sep 15 07:27:50.243342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:50.243310    2574 generic.go:356] "Generic (PLEG): container finished" podID="7544daa4-e98e-4d05-9040-007114388553" containerID="f61b070122df68b0352decb0fb358ba1e689036e5bc9ace6872785ea7ff05e30" exitCode=0
Sep 15 07:27:50.243733 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:50.243392    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn" event={"ID":"7544daa4-e98e-4d05-9040-007114388553","Type":"ContainerDied","Data":"f61b070122df68b0352decb0fb358ba1e689036e5bc9ace6872785ea7ff05e30"}
Sep 15 07:27:52.731748 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.731722    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:52.804169 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.804140    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kbq2\" (UniqueName: \"kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2\") pod \"7544daa4-e98e-4d05-9040-007114388553\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") "
Sep 15 07:27:52.804279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.804207    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util\") pod \"7544daa4-e98e-4d05-9040-007114388553\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") "
Sep 15 07:27:52.804279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.804263    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle\") pod \"7544daa4-e98e-4d05-9040-007114388553\" (UID: \"7544daa4-e98e-4d05-9040-007114388553\") "
Sep 15 07:27:52.804981 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.804937    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle" (OuterVolumeSpecName: "bundle") pod "7544daa4-e98e-4d05-9040-007114388553" (UID: "7544daa4-e98e-4d05-9040-007114388553"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:52.805945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.805921    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2" (OuterVolumeSpecName: "kube-api-access-6kbq2") pod "7544daa4-e98e-4d05-9040-007114388553" (UID: "7544daa4-e98e-4d05-9040-007114388553"). InnerVolumeSpecName "kube-api-access-6kbq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:27:52.808404 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.808381    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util" (OuterVolumeSpecName: "util") pod "7544daa4-e98e-4d05-9040-007114388553" (UID: "7544daa4-e98e-4d05-9040-007114388553"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:27:52.905618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.905595    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:52.905618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.905616    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbq2\" (UniqueName: \"kubernetes.io/projected/7544daa4-e98e-4d05-9040-007114388553-kube-api-access-6kbq2\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:52.905732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:52.905625    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7544daa4-e98e-4d05-9040-007114388553-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:27:53.258381 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:53.258348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8" event={"ID":"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520","Type":"ContainerStarted","Data":"f2f003cb5bc2f22b5361076760f68778d4c29a1ec114a49c53c613958323b87f"}
Sep 15 07:27:53.260097 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:53.260073    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn" event={"ID":"7544daa4-e98e-4d05-9040-007114388553","Type":"ContainerDied","Data":"30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230"}
Sep 15 07:27:53.260262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:53.260107    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cb0edc8a3f24515df8059ef2b818d5e9681122d1c4754c648bcc99c0870230"
Sep 15 07:27:53.260262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:53.260078    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0hq9fn"
Sep 15 07:27:54.267023 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:54.266988    2574 generic.go:356] "Generic (PLEG): container finished" podID="a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520" containerID="f2f003cb5bc2f22b5361076760f68778d4c29a1ec114a49c53c613958323b87f" exitCode=0
Sep 15 07:27:54.267486 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:54.267029    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8" event={"ID":"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520","Type":"ContainerDied","Data":"f2f003cb5bc2f22b5361076760f68778d4c29a1ec114a49c53c613958323b87f"}
Sep 15 07:27:55.274529 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:55.274489    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8" event={"ID":"a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520","Type":"ContainerStarted","Data":"a299daee3865f1c20dd465f956f911bb2bf41454e732f7bb9758e0b0dec3f97e"}
Sep 15 07:27:55.274938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:55.274598    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:27:55.297026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:27:55.296983    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8" podStartSLOduration=2.026397707 podStartE2EDuration="9.29696939s" totalImagesPullingTime="6.222951087s" totalInitContainerRuntime="1.047620596s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:27:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:27:55.29460294 +0000 UTC m=+563.388769202" watchObservedRunningTime="2026-09-15 07:27:55.29696939 +0000 UTC m=+563.391135652"
Sep 15 07:28:06.281246 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:06.281212    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-565d545f7b-tsjj8"
Sep 15 07:28:08.254536 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254503    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"]
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254738    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="util"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254748    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="util"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254756    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="extract"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254762    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="extract"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254775    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="pull"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254782    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="pull"
Sep 15 07:28:08.254878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.254820    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7544daa4-e98e-4d05-9040-007114388553" containerName="extract"
Sep 15 07:28:08.263250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.263229    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.265222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.265198    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 07:28:08.265741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.265718    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 07:28:08.265835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.265725    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 07:28:08.267363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.267342    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 07:28:08.273244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.273222    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"]
Sep 15 07:28:08.303861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.303836    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3536fa4f-d8f7-41e4-b371-a0687e74f611-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.303953 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.303873    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3536fa4f-d8f7-41e4-b371-a0687e74f611-tmp\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.303953 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.303893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbq4f\" (UniqueName: \"kubernetes.io/projected/3536fa4f-d8f7-41e4-b371-a0687e74f611-kube-api-access-cbq4f\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.404600 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.404575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3536fa4f-d8f7-41e4-b371-a0687e74f611-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.404698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.404612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3536fa4f-d8f7-41e4-b371-a0687e74f611-tmp\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.404698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.404631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbq4f\" (UniqueName: \"kubernetes.io/projected/3536fa4f-d8f7-41e4-b371-a0687e74f611-kube-api-access-cbq4f\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.406890 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.406870    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3536fa4f-d8f7-41e4-b371-a0687e74f611-tmp\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.407096 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.407078    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3536fa4f-d8f7-41e4-b371-a0687e74f611-tls-certs\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.413516 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.413497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbq4f\" (UniqueName: \"kubernetes.io/projected/3536fa4f-d8f7-41e4-b371-a0687e74f611-kube-api-access-cbq4f\") pod \"kube-auth-proxy-77b7bc76bc-v8gdn\" (UID: \"3536fa4f-d8f7-41e4-b371-a0687e74f611\") " pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.572635 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.572586    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"
Sep 15 07:28:08.839298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:08.839267    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn"]
Sep 15 07:28:08.841626 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:08.841605    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3536fa4f_d8f7_41e4_b371_a0687e74f611.slice/crio-ce3378da73868f438d73292510b7a212b043d038f1a0f37aabd1faaec06a357f WatchSource:0}: Error finding container ce3378da73868f438d73292510b7a212b043d038f1a0f37aabd1faaec06a357f: Status 404 returned error can't find the container with id ce3378da73868f438d73292510b7a212b043d038f1a0f37aabd1faaec06a357f
Sep 15 07:28:09.333618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.333579    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn" event={"ID":"3536fa4f-d8f7-41e4-b371-a0687e74f611","Type":"ContainerStarted","Data":"ce3378da73868f438d73292510b7a212b043d038f1a0f37aabd1faaec06a357f"}
Sep 15 07:28:09.471505 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.468834    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:09.474364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.474340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.476844 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.476819    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 07:28:09.476946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.476819    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 07:28:09.476946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.476850    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 07:28:09.476946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.476929    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fdlzs\""
Sep 15 07:28:09.476946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.476850    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 07:28:09.486726 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.486705    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:09.510884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.510855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511001 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.510918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.511051    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.511085    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.511151    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbvd\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.511170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.511268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.511242    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612608    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612650    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612689    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbvd\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612761    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612796    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.612870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.612833    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.613704 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.613653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.615871 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.615828    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.615953 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.615877    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.616318 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.616279    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.616813 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.616778    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.621070 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.621018    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.622889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.622850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbvd\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd\") pod \"istiod-openshift-gateway-7b7995d5cd-7hf78\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:09.783215 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:09.782961    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:10.074567 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:10.074539    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:10.077142 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:10.077089    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78dc40e_4bf5_4f0c_9b02_05461a0b49f2.slice/crio-e24b9a365f7d814940f870e3fd35d0f7115a0d672ca012404a0185ef118c65af WatchSource:0}: Error finding container e24b9a365f7d814940f870e3fd35d0f7115a0d672ca012404a0185ef118c65af: Status 404 returned error can't find the container with id e24b9a365f7d814940f870e3fd35d0f7115a0d672ca012404a0185ef118c65af
Sep 15 07:28:10.344257 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:10.344145    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" event={"ID":"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2","Type":"ContainerStarted","Data":"e24b9a365f7d814940f870e3fd35d0f7115a0d672ca012404a0185ef118c65af"}
Sep 15 07:28:12.355264 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:12.355226    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn" event={"ID":"3536fa4f-d8f7-41e4-b371-a0687e74f611","Type":"ContainerStarted","Data":"1f0bf8633a4218f6a4f6d200a9156c41eec45a65f4c642a2df80eaf5ae68856e"}
Sep 15 07:28:12.372355 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:12.372311    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-77b7bc76bc-v8gdn" podStartSLOduration=1.592194615 podStartE2EDuration="4.372292689s" totalImagesPullingTime="2.780098074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:12.371008515 +0000 UTC m=+580.465174777" watchObservedRunningTime="2026-09-15 07:28:12.372292689 +0000 UTC m=+580.466458951"
Sep 15 07:28:13.390265 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:13.390217    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:28:13.390659 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:13.390293    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:28:14.366167 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:14.366101    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" event={"ID":"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2","Type":"ContainerStarted","Data":"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"}
Sep 15 07:28:14.366379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:14.366280    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:14.367967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:14.367941    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:14.388384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:14.388313    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" podStartSLOduration=2.077729427 podStartE2EDuration="5.388297496s" totalImagesPullingTime="3.310568069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:14.38696635 +0000 UTC m=+582.481132611" watchObservedRunningTime="2026-09-15 07:28:14.388297496 +0000 UTC m=+582.482463758"
Sep 15 07:28:18.821543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.821504    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bkj9r"]
Sep 15 07:28:18.823714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.823694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:18.828586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.828563    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 15 07:28:18.828730 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.828713    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sp9qz\""
Sep 15 07:28:18.838835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.838816    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bkj9r"]
Sep 15 07:28:18.883661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.883636    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7g7\" (UniqueName: \"kubernetes.io/projected/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-kube-api-access-5h7g7\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:18.883778 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.883684    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:18.984966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.984934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7g7\" (UniqueName: \"kubernetes.io/projected/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-kube-api-access-5h7g7\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:18.985080 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.984984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:18.985168 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:18.985100    2574 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 15 07:28:18.985225 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:18.985169    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert podName:5e58c92a-eecb-4909-bf1b-b00b4059e4b9 nodeName:}" failed. No retries permitted until 2026-09-15 07:28:19.485152194 +0000 UTC m=+587.579318434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert") pod "odh-model-controller-7998c67dc4-bkj9r" (UID: "5e58c92a-eecb-4909-bf1b-b00b4059e4b9") : secret "odh-model-controller-webhook-cert" not found
Sep 15 07:28:18.992517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:18.992497    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7g7\" (UniqueName: \"kubernetes.io/projected/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-kube-api-access-5h7g7\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:19.488242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:19.488204    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:19.490752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:19.490724    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5e58c92a-eecb-4909-bf1b-b00b4059e4b9-cert\") pod \"odh-model-controller-7998c67dc4-bkj9r\" (UID: \"5e58c92a-eecb-4909-bf1b-b00b4059e4b9\") " pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:19.733579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:19.733556    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:19.999819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:19.999796    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bkj9r"]
Sep 15 07:28:20.002359 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:20.002325    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e58c92a_eecb_4909_bf1b_b00b4059e4b9.slice/crio-3ca9845868b509e371a45389da317c95b8d13b683abcf991cd0d54f8444ccf42 WatchSource:0}: Error finding container 3ca9845868b509e371a45389da317c95b8d13b683abcf991cd0d54f8444ccf42: Status 404 returned error can't find the container with id 3ca9845868b509e371a45389da317c95b8d13b683abcf991cd0d54f8444ccf42
Sep 15 07:28:20.391546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:20.391484    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r" event={"ID":"5e58c92a-eecb-4909-bf1b-b00b4059e4b9","Type":"ContainerStarted","Data":"3ca9845868b509e371a45389da317c95b8d13b683abcf991cd0d54f8444ccf42"}
Sep 15 07:28:23.410523 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:23.410487    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r" event={"ID":"5e58c92a-eecb-4909-bf1b-b00b4059e4b9","Type":"ContainerStarted","Data":"f66a83680e23fb68db0f15759ca6a96aab8638a399be1ecfd3d7a467c7fa7aca"}
Sep 15 07:28:23.410884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:23.410607    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:23.431156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:23.431088    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r" podStartSLOduration=2.421605847 podStartE2EDuration="5.431074178s" totalImagesPullingTime="3.009468331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:23.428886147 +0000 UTC m=+591.523052408" watchObservedRunningTime="2026-09-15 07:28:23.431074178 +0000 UTC m=+591.525240439"
Sep 15 07:28:28.689826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.689794    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"]
Sep 15 07:28:28.691886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.691863    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:28.694152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.694129    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 07:28:28.694910 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.694889    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-r6kjg\""
Sep 15 07:28:28.702347 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.702324    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"]
Sep 15 07:28:28.704322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.704307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.706428 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.706406    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 15 07:28:28.706532 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.706480    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jjrwp\""
Sep 15 07:28:28.708746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.708727    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"]
Sep 15 07:28:28.718513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.718491    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"]
Sep 15 07:28:28.854409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.854386    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6tnk\" (UniqueName: \"kubernetes.io/projected/cd7c9f17-47bc-4550-a393-c5aaa46943ee-kube-api-access-s6tnk\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:28.854517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.854422    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.854517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.854456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:28.854517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.854505    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mfh4\" (UniqueName: \"kubernetes.io/projected/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-kube-api-access-2mfh4\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.955138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.955063    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.955138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.955104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:28.955305 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.955149    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mfh4\" (UniqueName: \"kubernetes.io/projected/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-kube-api-access-2mfh4\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.955305 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.955184    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6tnk\" (UniqueName: \"kubernetes.io/projected/cd7c9f17-47bc-4550-a393-c5aaa46943ee-kube-api-access-s6tnk\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:28.955305 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:28.955236    2574 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 15 07:28:28.955449 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:28.955306    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert podName:cd7c9f17-47bc-4550-a393-c5aaa46943ee nodeName:}" failed. No retries permitted until 2026-09-15 07:28:29.455288255 +0000 UTC m=+597.549454495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert") pod "kserve-controller-manager-d6f4b65b8-smrhn" (UID: "cd7c9f17-47bc-4550-a393-c5aaa46943ee") : secret "kserve-webhook-server-cert" not found
Sep 15 07:28:28.955449 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:28.955240    2574 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 15 07:28:28.955449 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:28.955415    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert podName:54dbf0d5-437a-4355-8e8d-c2915d3aaa59 nodeName:}" failed. No retries permitted until 2026-09-15 07:28:29.455395592 +0000 UTC m=+597.549561836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert") pod "llmisvc-controller-manager-777c6bd7d5-q29mg" (UID: "54dbf0d5-437a-4355-8e8d-c2915d3aaa59") : secret "llmisvc-webhook-server-cert" not found
Sep 15 07:28:28.980109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.980081    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mfh4\" (UniqueName: \"kubernetes.io/projected/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-kube-api-access-2mfh4\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:28.981193 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:28.981176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6tnk\" (UniqueName: \"kubernetes.io/projected/cd7c9f17-47bc-4550-a393-c5aaa46943ee-kube-api-access-s6tnk\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:29.457815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.457789    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:29.457932 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.457841    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:29.460288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.460243    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd7c9f17-47bc-4550-a393-c5aaa46943ee-cert\") pod \"kserve-controller-manager-d6f4b65b8-smrhn\" (UID: \"cd7c9f17-47bc-4550-a393-c5aaa46943ee\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:29.460389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.460311    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54dbf0d5-437a-4355-8e8d-c2915d3aaa59-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-q29mg\" (UID: \"54dbf0d5-437a-4355-8e8d-c2915d3aaa59\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:29.601144 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.601104    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:29.614918 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.614898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:29.868826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.868800    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"]
Sep 15 07:28:29.870557 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:29.870533    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7c9f17_47bc_4550_a393_c5aaa46943ee.slice/crio-a64818379d90335d5d31f2cd2c0d506b08089ec8e55b3da2040b48e01aeb8cad WatchSource:0}: Error finding container a64818379d90335d5d31f2cd2c0d506b08089ec8e55b3da2040b48e01aeb8cad: Status 404 returned error can't find the container with id a64818379d90335d5d31f2cd2c0d506b08089ec8e55b3da2040b48e01aeb8cad
Sep 15 07:28:29.891338 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:29.891318    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"]
Sep 15 07:28:29.892949 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:29.892918    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod54dbf0d5_437a_4355_8e8d_c2915d3aaa59.slice/crio-6971384348dbc9367a41feefd3a2b66d05420f91e5021b34ce4b58d961469c4f WatchSource:0}: Error finding container 6971384348dbc9367a41feefd3a2b66d05420f91e5021b34ce4b58d961469c4f: Status 404 returned error can't find the container with id 6971384348dbc9367a41feefd3a2b66d05420f91e5021b34ce4b58d961469c4f
Sep 15 07:28:30.448718 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:30.448662    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg" event={"ID":"54dbf0d5-437a-4355-8e8d-c2915d3aaa59","Type":"ContainerStarted","Data":"6971384348dbc9367a41feefd3a2b66d05420f91e5021b34ce4b58d961469c4f"}
Sep 15 07:28:30.449980 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:30.449951    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn" event={"ID":"cd7c9f17-47bc-4550-a393-c5aaa46943ee","Type":"ContainerStarted","Data":"a64818379d90335d5d31f2cd2c0d506b08089ec8e55b3da2040b48e01aeb8cad"}
Sep 15 07:28:33.465658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.465616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg" event={"ID":"54dbf0d5-437a-4355-8e8d-c2915d3aaa59","Type":"ContainerStarted","Data":"1ba81604009bc1e749de6b791e7e26b3e6b104236032778615688098bec1fd98"}
Sep 15 07:28:33.466146 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.465729    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:28:33.470223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.470194    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn" event={"ID":"cd7c9f17-47bc-4550-a393-c5aaa46943ee","Type":"ContainerStarted","Data":"d2ad741e323d7511069f3f5d4dd72fc316e5ffccecf40cdeab44aa65519e8604"}
Sep 15 07:28:33.470359 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.470305    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:28:33.484914 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.484870    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg" podStartSLOduration=2.187392076 podStartE2EDuration="5.484856526s" totalImagesPullingTime="3.29746445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:33.484320679 +0000 UTC m=+601.578486941" watchObservedRunningTime="2026-09-15 07:28:33.484856526 +0000 UTC m=+601.579022810"
Sep 15 07:28:33.500529 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.500486    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn" podStartSLOduration=2.933156542 podStartE2EDuration="5.500472958s" totalImagesPullingTime="2.567316416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:33.49837102 +0000 UTC m=+601.592537283" watchObservedRunningTime="2026-09-15 07:28:33.500472958 +0000 UTC m=+601.594639220"
Sep 15 07:28:33.764876 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.764800    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:33.765005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:33.764984    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" podUID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" containerName="discovery" containerID="cri-o://597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc" gracePeriod=30
Sep 15 07:28:34.006170 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.006145    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:34.094546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094483    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094514    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmbvd\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094533    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094587    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094636    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094684    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.094740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.094715    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap\") pod \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\" (UID: \"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2\") "
Sep 15 07:28:34.095295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.095251    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:28:34.097053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097000    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs" (OuterVolumeSpecName: "local-certs") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:28:34.097053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097018    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:28:34.097230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts" (OuterVolumeSpecName: "cacerts") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:28:34.097230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097097    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd" (OuterVolumeSpecName: "kube-api-access-mmbvd") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "kube-api-access-mmbvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:28:34.097230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097204    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:28:34.097369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.097275    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" (UID: "d78dc40e-4bf5-4f0c-9b02-05461a0b49f2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:28:34.195539 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195514    2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-kubeconfig\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195539 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195536    2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-cacerts\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195546    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-dns-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195556    2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-csr-ca-configmap\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195565    2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-local-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195573    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbvd\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-kube-api-access-mmbvd\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.195655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.195581    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:28:34.420602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.420579    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bkj9r"
Sep 15 07:28:34.476672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.476643    2574 generic.go:356] "Generic (PLEG): container finished" podID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" containerID="597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc" exitCode=0
Sep 15 07:28:34.477088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.476709    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"
Sep 15 07:28:34.477088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.476768    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" event={"ID":"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2","Type":"ContainerDied","Data":"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"}
Sep 15 07:28:34.477088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.476806    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78" event={"ID":"d78dc40e-4bf5-4f0c-9b02-05461a0b49f2","Type":"ContainerDied","Data":"e24b9a365f7d814940f870e3fd35d0f7115a0d672ca012404a0185ef118c65af"}
Sep 15 07:28:34.477088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.476826    2574 scope.go:117] "RemoveContainer" containerID="597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"
Sep 15 07:28:34.486069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.486054    2574 scope.go:117] "RemoveContainer" containerID="597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"
Sep 15 07:28:34.486507 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:28:34.486475    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc\": container with ID starting with 597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc not found: ID does not exist" containerID="597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"
Sep 15 07:28:34.486595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.486519    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc"} err="failed to get container status \"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc\": rpc error: code = NotFound desc = could not find container \"597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc\": container with ID starting with 597eb7ea7827b8bd24c8a047a96ed831c6479cba5957e3937de62a3c49e229fc not found: ID does not exist"
Sep 15 07:28:34.494658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.494599    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:34.496819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:34.496798    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-7hf78"]
Sep 15 07:28:35.074836 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.074799    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5778f4bdf6-pmv7t"]
Sep 15 07:28:35.075136 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.075107    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" containerName="discovery"
Sep 15 07:28:35.075227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.075139    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" containerName="discovery"
Sep 15 07:28:35.075227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.075224    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" containerName="discovery"
Sep 15 07:28:35.079309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.079286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.082870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.082840    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 07:28:35.082950 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.082887    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 07:28:35.083926 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.083907    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 07:28:35.083926 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.083923    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 07:28:35.084101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.083973    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4rwg4\""
Sep 15 07:28:35.084212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.084198    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 07:28:35.084294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.084273    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 07:28:35.084354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.084291    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 07:28:35.088839 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.088818    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 07:28:35.090046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.090022    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5778f4bdf6-pmv7t"]
Sep 15 07:28:35.205921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.205896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-oauth-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.205921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.205928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-oauth-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.206145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.205949    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.206145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.205971    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.206145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.205986    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-service-ca\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.206145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.206025    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqvm\" (UniqueName: \"kubernetes.io/projected/75b9657e-4de1-4ec1-8bdd-5b849d270762-kube-api-access-csqvm\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.206145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.206052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-trusted-ca-bundle\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307297 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307267    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqvm\" (UniqueName: \"kubernetes.io/projected/75b9657e-4de1-4ec1-8bdd-5b849d270762-kube-api-access-csqvm\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307424 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-trusted-ca-bundle\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307508 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307484    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-oauth-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307563 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-oauth-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307611 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307577    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307619    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.307662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.307644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-service-ca\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.308266 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.308228    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-oauth-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.308414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.308393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.308479 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.308427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-service-ca\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.308479 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.308438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75b9657e-4de1-4ec1-8bdd-5b849d270762-trusted-ca-bundle\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.310065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.310043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-oauth-config\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.310159 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.310137    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/75b9657e-4de1-4ec1-8bdd-5b849d270762-console-serving-cert\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.317430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.317412    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqvm\" (UniqueName: \"kubernetes.io/projected/75b9657e-4de1-4ec1-8bdd-5b849d270762-kube-api-access-csqvm\") pod \"console-5778f4bdf6-pmv7t\" (UID: \"75b9657e-4de1-4ec1-8bdd-5b849d270762\") " pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.389130 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.389055    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:35.659083 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:35.659056    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5778f4bdf6-pmv7t"]
Sep 15 07:28:35.661300 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:28:35.661277    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b9657e_4de1_4ec1_8bdd_5b849d270762.slice/crio-a2527677a7b9c36d4813044ed7c2de2fc94fa59d157880912edfde4a9867cbdd WatchSource:0}: Error finding container a2527677a7b9c36d4813044ed7c2de2fc94fa59d157880912edfde4a9867cbdd: Status 404 returned error can't find the container with id a2527677a7b9c36d4813044ed7c2de2fc94fa59d157880912edfde4a9867cbdd
Sep 15 07:28:36.378690 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:36.378656    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78dc40e-4bf5-4f0c-9b02-05461a0b49f2" path="/var/lib/kubelet/pods/d78dc40e-4bf5-4f0c-9b02-05461a0b49f2/volumes"
Sep 15 07:28:36.492249 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:36.492215    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5778f4bdf6-pmv7t" event={"ID":"75b9657e-4de1-4ec1-8bdd-5b849d270762","Type":"ContainerStarted","Data":"471677f6b986e67207545101f2782e1e9c115a21bfb7348579777d9d7c32b87a"}
Sep 15 07:28:36.492249 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:36.492257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5778f4bdf6-pmv7t" event={"ID":"75b9657e-4de1-4ec1-8bdd-5b849d270762","Type":"ContainerStarted","Data":"a2527677a7b9c36d4813044ed7c2de2fc94fa59d157880912edfde4a9867cbdd"}
Sep 15 07:28:36.509094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:36.509042    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5778f4bdf6-pmv7t" podStartSLOduration=1.509026051 podStartE2EDuration="1.509026051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:28:36.507277754 +0000 UTC m=+604.601444043" watchObservedRunningTime="2026-09-15 07:28:36.509026051 +0000 UTC m=+604.603192313"
Sep 15 07:28:45.389349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:45.389310    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:45.389916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:45.389403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:45.394173 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:45.394149    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:28:45.534861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:28:45.534839    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5778f4bdf6-pmv7t"
Sep 15 07:29:04.478879 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:04.478792    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-q29mg"
Sep 15 07:29:04.482779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:04.482758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-smrhn"
Sep 15 07:29:23.624677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.624635    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"]
Sep 15 07:29:23.631461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.631442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.633920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.633895    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 07:29:23.633920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.633910    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 07:29:23.634499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.634484    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nftv2\""
Sep 15 07:29:23.638234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.638215    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"]
Sep 15 07:29:23.723285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.723251    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"]
Sep 15 07:29:23.726467 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.726451    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.734546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.734522    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"]
Sep 15 07:29:23.741439 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.741417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.741534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.741456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.741534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.741480    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrhjh\" (UniqueName: \"kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.823407 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.823375    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"]
Sep 15 07:29:23.826701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.826683    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:23.834533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.834510    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"]
Sep 15 07:29:23.841979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.841956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrhjh\" (UniqueName: \"kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.842096 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842004    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.842096 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842041    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.842096 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842073    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824k6\" (UniqueName: \"kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.842227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.842302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842274    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.842618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842597    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.842727 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.842622    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.849793 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.849767    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrhjh\" (UniqueName: \"kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.926217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.926190    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"]
Sep 15 07:29:23.929491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.929476    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:23.937753 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.937726    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"]
Sep 15 07:29:23.941617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.941593    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:23.942852 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.942831    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:23.942941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.942867    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:23.942941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.942893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.942941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.942926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.943061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.942956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824k6\" (UniqueName: \"kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.943061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.943010    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wlk\" (UniqueName: \"kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:23.943273 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.943255    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.943332 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.943274    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:23.951707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:23.951676    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824k6\" (UniqueName: \"kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:24.035546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.035503    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:24.043654 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.043767 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043664    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w5ln\" (UniqueName: \"kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.043767 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.043902 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043883    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.043963 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.044014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.043982    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.044066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.044024    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.044066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.044040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wlk\" (UniqueName: \"kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.053314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.053289    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wlk\" (UniqueName: \"kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.136704 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.136672    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:24.144821 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.144802    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.144898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.144854    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w5ln\" (UniqueName: \"kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.144995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.144974    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.145215 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.145196    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.145295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.145277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.154564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.154542    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w5ln\" (UniqueName: \"kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.214329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.214300    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"]
Sep 15 07:29:24.215992 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:24.215968    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bb1ba2_c64e_4fc9_8477_aa611b77841d.slice/crio-b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183 WatchSource:0}: Error finding container b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183: Status 404 returned error can't find the container with id b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183
Sep 15 07:29:24.239267 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.239245    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:24.307870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.307845    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"]
Sep 15 07:29:24.311718 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:24.311684    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe188d6f_9ea9_4e50_9d05_8f85c8e55158.slice/crio-5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad WatchSource:0}: Error finding container 5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad: Status 404 returned error can't find the container with id 5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad
Sep 15 07:29:24.404620 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.404595    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"]
Sep 15 07:29:24.406179 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:24.406156    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6685f2b5_b02a_4826_b798_df9bbaea9ca4.slice/crio-46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14 WatchSource:0}: Error finding container 46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14: Status 404 returned error can't find the container with id 46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14
Sep 15 07:29:24.508938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.508914    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"]
Sep 15 07:29:24.510577 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:24.510552    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409001b5_60ec_4fb8_98ad_000790db4281.slice/crio-14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202 WatchSource:0}: Error finding container 14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202: Status 404 returned error can't find the container with id 14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202
Sep 15 07:29:24.699663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.699608    2574 generic.go:356] "Generic (PLEG): container finished" podID="409001b5-60ec-4fb8-98ad-000790db4281" containerID="544644b22154abf913ad357419a6def96982b52030d4feae45e721b47c8a01be" exitCode=0
Sep 15 07:29:24.700002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.699684    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm" event={"ID":"409001b5-60ec-4fb8-98ad-000790db4281","Type":"ContainerDied","Data":"544644b22154abf913ad357419a6def96982b52030d4feae45e721b47c8a01be"}
Sep 15 07:29:24.700002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.699732    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm" event={"ID":"409001b5-60ec-4fb8-98ad-000790db4281","Type":"ContainerStarted","Data":"14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202"}
Sep 15 07:29:24.700680 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.700663    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:29:24.701346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.701324    2574 generic.go:356] "Generic (PLEG): container finished" podID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerID="33bfe189607748b0d6054fa6ceeedec01abac17083bd758d51c7a5857833b649" exitCode=0
Sep 15 07:29:24.701480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.701458    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4" event={"ID":"6685f2b5-b02a-4826-b798-df9bbaea9ca4","Type":"ContainerDied","Data":"33bfe189607748b0d6054fa6ceeedec01abac17083bd758d51c7a5857833b649"}
Sep 15 07:29:24.701611 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.701508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4" event={"ID":"6685f2b5-b02a-4826-b798-df9bbaea9ca4","Type":"ContainerStarted","Data":"46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14"}
Sep 15 07:29:24.703058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.703034    2574 generic.go:356] "Generic (PLEG): container finished" podID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerID="b4dfc5ae93707101e79bf095a1e5f39f77a6e04cb0ba2b3c2b52813d82609ebe" exitCode=0
Sep 15 07:29:24.703184 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.703091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl" event={"ID":"be188d6f-9ea9-4e50-9d05-8f85c8e55158","Type":"ContainerDied","Data":"b4dfc5ae93707101e79bf095a1e5f39f77a6e04cb0ba2b3c2b52813d82609ebe"}
Sep 15 07:29:24.703184 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.703152    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl" event={"ID":"be188d6f-9ea9-4e50-9d05-8f85c8e55158","Type":"ContainerStarted","Data":"5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad"}
Sep 15 07:29:24.704831 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.704781    2574 generic.go:356] "Generic (PLEG): container finished" podID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerID="a1be43acd322a7927dd0b02858e04eeb955913a408c1e26400eeb7c070e8db45" exitCode=0
Sep 15 07:29:24.704915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.704834    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5" event={"ID":"24bb1ba2-c64e-4fc9-8477-aa611b77841d","Type":"ContainerDied","Data":"a1be43acd322a7927dd0b02858e04eeb955913a408c1e26400eeb7c070e8db45"}
Sep 15 07:29:24.704915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:24.704879    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5" event={"ID":"24bb1ba2-c64e-4fc9-8477-aa611b77841d","Type":"ContainerStarted","Data":"b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183"}
Sep 15 07:29:26.715577 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.715542    2574 generic.go:356] "Generic (PLEG): container finished" podID="409001b5-60ec-4fb8-98ad-000790db4281" containerID="581a0c9bcb58172c970933b4f0057376db06007697a47ecbba6697632e96364b" exitCode=0
Sep 15 07:29:26.715976 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.715616    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm" event={"ID":"409001b5-60ec-4fb8-98ad-000790db4281","Type":"ContainerDied","Data":"581a0c9bcb58172c970933b4f0057376db06007697a47ecbba6697632e96364b"}
Sep 15 07:29:26.717582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.717554    2574 generic.go:356] "Generic (PLEG): container finished" podID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerID="8318fe7be763f07e3b5239c287faec5710df4ad64432f8634dcf9f5c62928240" exitCode=0
Sep 15 07:29:26.717684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.717641    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4" event={"ID":"6685f2b5-b02a-4826-b798-df9bbaea9ca4","Type":"ContainerDied","Data":"8318fe7be763f07e3b5239c287faec5710df4ad64432f8634dcf9f5c62928240"}
Sep 15 07:29:26.722200 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.722171    2574 generic.go:356] "Generic (PLEG): container finished" podID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerID="3cf188713b1689c198591c2868fcc69224c244f030991ea4f16168780599c257" exitCode=0
Sep 15 07:29:26.722306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.722222    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl" event={"ID":"be188d6f-9ea9-4e50-9d05-8f85c8e55158","Type":"ContainerDied","Data":"3cf188713b1689c198591c2868fcc69224c244f030991ea4f16168780599c257"}
Sep 15 07:29:26.724359 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.724337    2574 generic.go:356] "Generic (PLEG): container finished" podID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerID="47c937a98d19b31c8b10af378dcbf95b8bb1fd0b030b1bc0078f28c766a95e57" exitCode=0
Sep 15 07:29:26.724447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:26.724382    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5" event={"ID":"24bb1ba2-c64e-4fc9-8477-aa611b77841d","Type":"ContainerDied","Data":"47c937a98d19b31c8b10af378dcbf95b8bb1fd0b030b1bc0078f28c766a95e57"}
Sep 15 07:29:27.732863 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.732821    2574 generic.go:356] "Generic (PLEG): container finished" podID="409001b5-60ec-4fb8-98ad-000790db4281" containerID="afecd395c211eca4e4324b52846d50ae1a0d10085580714dbd5ca128ad27d3e6" exitCode=0
Sep 15 07:29:27.733303 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.732909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm" event={"ID":"409001b5-60ec-4fb8-98ad-000790db4281","Type":"ContainerDied","Data":"afecd395c211eca4e4324b52846d50ae1a0d10085580714dbd5ca128ad27d3e6"}
Sep 15 07:29:27.735593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.735567    2574 generic.go:356] "Generic (PLEG): container finished" podID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerID="6c45be9637e160a26bcf80aff5db6e64515fb80162b0d79dc891dfd928ee7433" exitCode=0
Sep 15 07:29:27.735739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.735609    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4" event={"ID":"6685f2b5-b02a-4826-b798-df9bbaea9ca4","Type":"ContainerDied","Data":"6c45be9637e160a26bcf80aff5db6e64515fb80162b0d79dc891dfd928ee7433"}
Sep 15 07:29:27.738429 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.738404    2574 generic.go:356] "Generic (PLEG): container finished" podID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerID="826dba24bd6534bdcd5f07aff94f635b2a5945f5cda172791b69cc770dc8face" exitCode=0
Sep 15 07:29:27.738541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.738479    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl" event={"ID":"be188d6f-9ea9-4e50-9d05-8f85c8e55158","Type":"ContainerDied","Data":"826dba24bd6534bdcd5f07aff94f635b2a5945f5cda172791b69cc770dc8face"}
Sep 15 07:29:27.740781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.740756    2574 generic.go:356] "Generic (PLEG): container finished" podID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerID="746b7b0e30c0a068fedd04546f4eaedd48da1aeb0274d02bc6160d76a50748c7" exitCode=0
Sep 15 07:29:27.740880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:27.740787    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5" event={"ID":"24bb1ba2-c64e-4fc9-8477-aa611b77841d","Type":"ContainerDied","Data":"746b7b0e30c0a068fedd04546f4eaedd48da1aeb0274d02bc6160d76a50748c7"}
Sep 15 07:29:28.891103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.891071    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:28.917499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.917474    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:28.936147 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.936125    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:28.940368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.940348    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:28.985770 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.985745    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util\") pod \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") "
Sep 15 07:29:28.985883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.985794    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2w5ln\" (UniqueName: \"kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln\") pod \"409001b5-60ec-4fb8-98ad-000790db4281\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") "
Sep 15 07:29:28.985883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.985866    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util\") pod \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") "
Sep 15 07:29:28.986007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.985891    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrhjh\" (UniqueName: \"kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh\") pod \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") "
Sep 15 07:29:28.986007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.985910    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util\") pod \"409001b5-60ec-4fb8-98ad-000790db4281\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") "
Sep 15 07:29:28.986090 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986070    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle\") pod \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") "
Sep 15 07:29:28.986140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986109    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6wlk\" (UniqueName: \"kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk\") pod \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\" (UID: \"6685f2b5-b02a-4826-b798-df9bbaea9ca4\") "
Sep 15 07:29:28.986198 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986160    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle\") pod \"409001b5-60ec-4fb8-98ad-000790db4281\" (UID: \"409001b5-60ec-4fb8-98ad-000790db4281\") "
Sep 15 07:29:28.986248 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986195    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util\") pod \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") "
Sep 15 07:29:28.986248 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986218    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle\") pod \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") "
Sep 15 07:29:28.986347 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986273    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824k6\" (UniqueName: \"kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6\") pod \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\" (UID: \"be188d6f-9ea9-4e50-9d05-8f85c8e55158\") "
Sep 15 07:29:28.986347 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.986307    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle\") pod \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\" (UID: \"24bb1ba2-c64e-4fc9-8477-aa611b77841d\") "
Sep 15 07:29:28.988587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.987659    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle" (OuterVolumeSpecName: "bundle") pod "6685f2b5-b02a-4826-b798-df9bbaea9ca4" (UID: "6685f2b5-b02a-4826-b798-df9bbaea9ca4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.988587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.987731    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle" (OuterVolumeSpecName: "bundle") pod "be188d6f-9ea9-4e50-9d05-8f85c8e55158" (UID: "be188d6f-9ea9-4e50-9d05-8f85c8e55158"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.988587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.988526    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle" (OuterVolumeSpecName: "bundle") pod "409001b5-60ec-4fb8-98ad-000790db4281" (UID: "409001b5-60ec-4fb8-98ad-000790db4281"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.988587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.988567    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle" (OuterVolumeSpecName: "bundle") pod "24bb1ba2-c64e-4fc9-8477-aa611b77841d" (UID: "24bb1ba2-c64e-4fc9-8477-aa611b77841d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.989305 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.989279    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln" (OuterVolumeSpecName: "kube-api-access-2w5ln") pod "409001b5-60ec-4fb8-98ad-000790db4281" (UID: "409001b5-60ec-4fb8-98ad-000790db4281"). InnerVolumeSpecName "kube-api-access-2w5ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:28.989404 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.989308    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh" (OuterVolumeSpecName: "kube-api-access-jrhjh") pod "24bb1ba2-c64e-4fc9-8477-aa611b77841d" (UID: "24bb1ba2-c64e-4fc9-8477-aa611b77841d"). InnerVolumeSpecName "kube-api-access-jrhjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:28.989404 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.989339    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk" (OuterVolumeSpecName: "kube-api-access-t6wlk") pod "6685f2b5-b02a-4826-b798-df9bbaea9ca4" (UID: "6685f2b5-b02a-4826-b798-df9bbaea9ca4"). InnerVolumeSpecName "kube-api-access-t6wlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:28.991004 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.990978    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6" (OuterVolumeSpecName: "kube-api-access-824k6") pod "be188d6f-9ea9-4e50-9d05-8f85c8e55158" (UID: "be188d6f-9ea9-4e50-9d05-8f85c8e55158"). InnerVolumeSpecName "kube-api-access-824k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:28.992521 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.992501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util" (OuterVolumeSpecName: "util") pod "6685f2b5-b02a-4826-b798-df9bbaea9ca4" (UID: "6685f2b5-b02a-4826-b798-df9bbaea9ca4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.993275 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.993255    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util" (OuterVolumeSpecName: "util") pod "409001b5-60ec-4fb8-98ad-000790db4281" (UID: "409001b5-60ec-4fb8-98ad-000790db4281"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.993633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.993616    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util" (OuterVolumeSpecName: "util") pod "be188d6f-9ea9-4e50-9d05-8f85c8e55158" (UID: "be188d6f-9ea9-4e50-9d05-8f85c8e55158"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:28.993820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:28.993802    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util" (OuterVolumeSpecName: "util") pod "24bb1ba2-c64e-4fc9-8477-aa611b77841d" (UID: "24bb1ba2-c64e-4fc9-8477-aa611b77841d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:29.087515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087465    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087485    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2w5ln\" (UniqueName: \"kubernetes.io/projected/409001b5-60ec-4fb8-98ad-000790db4281-kube-api-access-2w5ln\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087495    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087504    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrhjh\" (UniqueName: \"kubernetes.io/projected/24bb1ba2-c64e-4fc9-8477-aa611b77841d-kube-api-access-jrhjh\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087511    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087519    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6685f2b5-b02a-4826-b798-df9bbaea9ca4-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087528    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wlk\" (UniqueName: \"kubernetes.io/projected/6685f2b5-b02a-4826-b798-df9bbaea9ca4-kube-api-access-t6wlk\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087536    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/409001b5-60ec-4fb8-98ad-000790db4281-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087543    2574 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-util\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087550    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/be188d6f-9ea9-4e50-9d05-8f85c8e55158-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087558    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824k6\" (UniqueName: \"kubernetes.io/projected/be188d6f-9ea9-4e50-9d05-8f85c8e55158-kube-api-access-824k6\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.087719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.087565    2574 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24bb1ba2-c64e-4fc9-8477-aa611b77841d-bundle\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:29.752732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.752706    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl"
Sep 15 07:29:29.752732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.752718    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4czfwnl" event={"ID":"be188d6f-9ea9-4e50-9d05-8f85c8e55158","Type":"ContainerDied","Data":"5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad"}
Sep 15 07:29:29.752939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.752766    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f2cf4cd8be91afe33f89b7730262ec2ae9f251d14084c7ee503bd0ebb3021ad"
Sep 15 07:29:29.754999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.754975    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5"
Sep 15 07:29:29.755156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.755003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hckw5" event={"ID":"24bb1ba2-c64e-4fc9-8477-aa611b77841d","Type":"ContainerDied","Data":"b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183"}
Sep 15 07:29:29.755156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.755044    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1260988f35a9ed321f89a7d3f31d326690d64a7ebbbf9c3e54f5abeffc9b183"
Sep 15 07:29:29.757302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.757285    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm"
Sep 15 07:29:29.757397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.757299    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rnpbm" event={"ID":"409001b5-60ec-4fb8-98ad-000790db4281","Type":"ContainerDied","Data":"14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202"}
Sep 15 07:29:29.757397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.757340    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14f642cfd06a8a90c20e07f2ca510c7d21ebe445299f456d5b6e9784da8c9202"
Sep 15 07:29:29.759587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.759559    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4" event={"ID":"6685f2b5-b02a-4826-b798-df9bbaea9ca4","Type":"ContainerDied","Data":"46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14"}
Sep 15 07:29:29.759693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.759608    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46064c474ba33a64ae323602c7d585db227f365ba327183df5ce2beaf038ff14"
Sep 15 07:29:29.759693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:29.759566    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847axzbd4"
Sep 15 07:29:36.611708 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611672    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"]
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611936    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611946    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611959    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611964    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611973    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611979    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611987    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.611992    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612000    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612005    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612013    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612018    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612024    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612029    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="util"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612035    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612040    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612049    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612054    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612060    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612066    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612071    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612076    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="pull"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612081    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="extract"
Sep 15 07:29:36.612089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612086    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="extract"
Sep 15 07:29:36.612858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612145    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6685f2b5-b02a-4826-b798-df9bbaea9ca4" containerName="extract"
Sep 15 07:29:36.612858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612152    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="24bb1ba2-c64e-4fc9-8477-aa611b77841d" containerName="extract"
Sep 15 07:29:36.612858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612158    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="409001b5-60ec-4fb8-98ad-000790db4281" containerName="extract"
Sep 15 07:29:36.612858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.612164    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="be188d6f-9ea9-4e50-9d05-8f85c8e55158" containerName="extract"
Sep 15 07:29:36.616832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.616812    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:36.619178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.619152    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 07:29:36.619340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.619195    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gxxpj\""
Sep 15 07:29:36.619340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.619226    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 07:29:36.619340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.619230    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 07:29:36.624885 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.624860    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"]
Sep 15 07:29:36.741020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.740991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r69bx\" (UniqueName: \"kubernetes.io/projected/08aeeb1f-4086-4b9e-84df-61f6d93a3534-kube-api-access-r69bx\") pod \"dns-operator-controller-manager-7b84cd7c9c-r9tjx\" (UID: \"08aeeb1f-4086-4b9e-84df-61f6d93a3534\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:36.841702 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.841673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r69bx\" (UniqueName: \"kubernetes.io/projected/08aeeb1f-4086-4b9e-84df-61f6d93a3534-kube-api-access-r69bx\") pod \"dns-operator-controller-manager-7b84cd7c9c-r9tjx\" (UID: \"08aeeb1f-4086-4b9e-84df-61f6d93a3534\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:36.868212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.868155    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r69bx\" (UniqueName: \"kubernetes.io/projected/08aeeb1f-4086-4b9e-84df-61f6d93a3534-kube-api-access-r69bx\") pod \"dns-operator-controller-manager-7b84cd7c9c-r9tjx\" (UID: \"08aeeb1f-4086-4b9e-84df-61f6d93a3534\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:36.928068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:36.928034    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:37.205536 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:37.205513    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"]
Sep 15 07:29:37.207247 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:37.207224    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08aeeb1f_4086_4b9e_84df_61f6d93a3534.slice/crio-4c0ea2827bf9e2d68f79a224cc9ac5c868c39bc35909c1b1fc92df6d86f34b41 WatchSource:0}: Error finding container 4c0ea2827bf9e2d68f79a224cc9ac5c868c39bc35909c1b1fc92df6d86f34b41: Status 404 returned error can't find the container with id 4c0ea2827bf9e2d68f79a224cc9ac5c868c39bc35909c1b1fc92df6d86f34b41
Sep 15 07:29:37.796200 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:37.796161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx" event={"ID":"08aeeb1f-4086-4b9e-84df-61f6d93a3534","Type":"ContainerStarted","Data":"4c0ea2827bf9e2d68f79a224cc9ac5c868c39bc35909c1b1fc92df6d86f34b41"}
Sep 15 07:29:39.809263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:39.809164    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx" event={"ID":"08aeeb1f-4086-4b9e-84df-61f6d93a3534","Type":"ContainerStarted","Data":"157de275a640dd2f85a112c15b40234fe7ab25ece3b268382acc7331a392623b"}
Sep 15 07:29:39.809263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:39.809220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:39.835774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:39.835726    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx" podStartSLOduration=1.493820299 podStartE2EDuration="3.835710925s" totalImagesPullingTime="2.341890626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:39.834501056 +0000 UTC m=+667.928667319" watchObservedRunningTime="2026-09-15 07:29:39.835710925 +0000 UTC m=+667.929877186"
Sep 15 07:29:40.787012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.786977    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"]
Sep 15 07:29:40.790248 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.790232    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.793444 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.793422    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5dx5m\""
Sep 15 07:29:40.808662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.808631    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"]
Sep 15 07:29:40.875774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.875748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.876056 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.875795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd62j\" (UniqueName: \"kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.976483 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.976458    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd62j\" (UniqueName: \"kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.976573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.976519    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.976834 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.976819    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:40.984794 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:40.984773    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd62j\" (UniqueName: \"kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:41.100376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:41.100323    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:41.390078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:41.390053    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"]
Sep 15 07:29:41.391361 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:41.391332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3212025f_ad33_4c26_b843_9417923b6d0f.slice/crio-32126570034a420224a896fd122c0c23bc4f99f7670da8c88e45f731a9177179 WatchSource:0}: Error finding container 32126570034a420224a896fd122c0c23bc4f99f7670da8c88e45f731a9177179: Status 404 returned error can't find the container with id 32126570034a420224a896fd122c0c23bc4f99f7670da8c88e45f731a9177179
Sep 15 07:29:41.819482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:41.819455    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8" event={"ID":"3212025f-ad33-4c26-b843-9417923b6d0f","Type":"ContainerStarted","Data":"32126570034a420224a896fd122c0c23bc4f99f7670da8c88e45f731a9177179"}
Sep 15 07:29:43.386042 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.386001    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cx79r"]
Sep 15 07:29:43.389665 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.389644    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:43.391696 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.391632    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-swr9q\""
Sep 15 07:29:43.402430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.402407    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cx79r"]
Sep 15 07:29:43.496977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.496864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj46h\" (UniqueName: \"kubernetes.io/projected/c48d52a2-e97c-4db6-b960-c48cf826201c-kube-api-access-sj46h\") pod \"authorino-operator-76b478bd78-cx79r\" (UID: \"c48d52a2-e97c-4db6-b960-c48cf826201c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:43.597764 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.597733    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj46h\" (UniqueName: \"kubernetes.io/projected/c48d52a2-e97c-4db6-b960-c48cf826201c-kube-api-access-sj46h\") pod \"authorino-operator-76b478bd78-cx79r\" (UID: \"c48d52a2-e97c-4db6-b960-c48cf826201c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:43.608460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.608438    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj46h\" (UniqueName: \"kubernetes.io/projected/c48d52a2-e97c-4db6-b960-c48cf826201c-kube-api-access-sj46h\") pod \"authorino-operator-76b478bd78-cx79r\" (UID: \"c48d52a2-e97c-4db6-b960-c48cf826201c\") " pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:43.702235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.702210    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:43.999086 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:43.999056    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-cx79r"]
Sep 15 07:29:44.981535 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:44.981502    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48d52a2_e97c_4db6_b960_c48cf826201c.slice/crio-f54712be43bf4f6c86045da68ab5dd3e79a3d25f7a78452b35e097c62ecac205 WatchSource:0}: Error finding container f54712be43bf4f6c86045da68ab5dd3e79a3d25f7a78452b35e097c62ecac205: Status 404 returned error can't find the container with id f54712be43bf4f6c86045da68ab5dd3e79a3d25f7a78452b35e097c62ecac205
Sep 15 07:29:45.591146 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.591102    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"]
Sep 15 07:29:45.594349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.594334    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:45.596728 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.596695    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nn6rf\""
Sep 15 07:29:45.606305 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.606282    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"]
Sep 15 07:29:45.713197 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.713169    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qzt\" (UniqueName: \"kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt\") pod \"limitador-operator-controller-manager-74495c69f8-9c9fn\" (UID: \"da1a6f97-c508-486b-832f-2432c140deb3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:45.814469 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.814438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qzt\" (UniqueName: \"kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt\") pod \"limitador-operator-controller-manager-74495c69f8-9c9fn\" (UID: \"da1a6f97-c508-486b-832f-2432c140deb3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:45.824880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.824853    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qzt\" (UniqueName: \"kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt\") pod \"limitador-operator-controller-manager-74495c69f8-9c9fn\" (UID: \"da1a6f97-c508-486b-832f-2432c140deb3\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:45.841891 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.841829    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8" event={"ID":"3212025f-ad33-4c26-b843-9417923b6d0f","Type":"ContainerStarted","Data":"49ffb9740aeb15bec88af089fe5f5415bf20eed9180c978576a624a2979e6313"}
Sep 15 07:29:45.841996 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.841981    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:45.843107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.843088    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r" event={"ID":"c48d52a2-e97c-4db6-b960-c48cf826201c","Type":"ContainerStarted","Data":"f54712be43bf4f6c86045da68ab5dd3e79a3d25f7a78452b35e097c62ecac205"}
Sep 15 07:29:45.862579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.862535    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8" podStartSLOduration=2.205328573 podStartE2EDuration="5.862520019s" totalImagesPullingTime="3.657191446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:45.85987988 +0000 UTC m=+673.954046139" watchObservedRunningTime="2026-09-15 07:29:45.862520019 +0000 UTC m=+673.956686296"
Sep 15 07:29:45.903847 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:45.903827    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:46.173543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:46.173489    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"]
Sep 15 07:29:46.176382 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:46.176346    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1a6f97_c508_486b_832f_2432c140deb3.slice/crio-3da510a634d3478c43032d3276c12d8ddc8cca965349d6b6edb54feb188ce856 WatchSource:0}: Error finding container 3da510a634d3478c43032d3276c12d8ddc8cca965349d6b6edb54feb188ce856: Status 404 returned error can't find the container with id 3da510a634d3478c43032d3276c12d8ddc8cca965349d6b6edb54feb188ce856
Sep 15 07:29:46.849853 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:46.849809    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn" event={"ID":"da1a6f97-c508-486b-832f-2432c140deb3","Type":"ContainerStarted","Data":"3da510a634d3478c43032d3276c12d8ddc8cca965349d6b6edb54feb188ce856"}
Sep 15 07:29:46.851803 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:46.851772    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r" event={"ID":"c48d52a2-e97c-4db6-b960-c48cf826201c","Type":"ContainerStarted","Data":"1ac498e0971c3de1da128dcdf4f5ce2543b68dc15bf627622d84fb63673bcc2b"}
Sep 15 07:29:46.851946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:46.851838    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:46.875550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:46.875507    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r" podStartSLOduration=2.182230945 podStartE2EDuration="3.87549214s" totalImagesPullingTime="1.693261195s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:46.872331265 +0000 UTC m=+674.966497528" watchObservedRunningTime="2026-09-15 07:29:46.87549214 +0000 UTC m=+674.969658401"
Sep 15 07:29:48.863946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:48.863909    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn" event={"ID":"da1a6f97-c508-486b-832f-2432c140deb3","Type":"ContainerStarted","Data":"6c0d6c64ac2ac548398bc258b4a65859755ac09e95c5f6f6f43d2f8f683ed909"}
Sep 15 07:29:48.864352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:48.864049    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:48.880140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:48.880067    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn" podStartSLOduration=1.8466756389999999 podStartE2EDuration="3.88005305s" totalImagesPullingTime="2.033377411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:48.87876779 +0000 UTC m=+676.972934053" watchObservedRunningTime="2026-09-15 07:29:48.88005305 +0000 UTC m=+676.974219312"
Sep 15 07:29:50.816028 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:50.815999    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-r9tjx"
Sep 15 07:29:56.853907 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:56.853871    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:57.859382 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:57.859337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-cx79r"
Sep 15 07:29:58.753388 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.753343    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"]
Sep 15 07:29:58.753605 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.753557    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" containerName="manager" containerID="cri-o://49ffb9740aeb15bec88af089fe5f5415bf20eed9180c978576a624a2979e6313" gracePeriod=2
Sep 15 07:29:58.764573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.764545    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"]
Sep 15 07:29:58.777351 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:58.777307    2574 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"3212025f-ad33-4c26-b843-9417923b6d0f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\": pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:58.779089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.779057    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:29:58.779562 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.779538    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" containerName="manager"
Sep 15 07:29:58.779630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.779567    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" containerName="manager"
Sep 15 07:29:58.779669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.779643    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" containerName="manager"
Sep 15 07:29:58.781999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.781978    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:58.783742 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.783718    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"]
Sep 15 07:29:58.783988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.783942    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn" podUID="da1a6f97-c508-486b-832f-2432c140deb3" containerName="manager" containerID="cri-o://6c0d6c64ac2ac548398bc258b4a65859755ac09e95c5f6f6f43d2f8f683ed909" gracePeriod=2
Sep 15 07:29:58.785876 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.785854    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:58.797835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.797813    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:29:58.800374 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.800349    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"]
Sep 15 07:29:58.825537 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:58.825508    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:58.827282 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:58.827249    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:58.828929 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:58.828901    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:58.914889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.914851    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:58.915288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.914915    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42798\" (UniqueName: \"kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:58.922780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.922416    2574 generic.go:356] "Generic (PLEG): container finished" podID="3212025f-ad33-4c26-b843-9417923b6d0f" containerID="49ffb9740aeb15bec88af089fe5f5415bf20eed9180c978576a624a2979e6313" exitCode=0
Sep 15 07:29:58.924444 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:58.924425    2574 generic.go:356] "Generic (PLEG): container finished" podID="da1a6f97-c508-486b-832f-2432c140deb3" containerID="6c0d6c64ac2ac548398bc258b4a65859755ac09e95c5f6f6f43d2f8f683ed909" exitCode=0
Sep 15 07:29:59.015403 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.015373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42798\" (UniqueName: \"kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.015500 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.015460    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.015809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.015788    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.018481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.018460    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.020440 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.020416    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.021196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.021178    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.022146 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.022104    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.023873 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.023852    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.025358 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.025339    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.026709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.026690    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42798\" (UniqueName: \"kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798\") pod \"kuadrant-operator-controller-manager-76f8f686bf-d9s2k\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.115709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.115683    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pd62j\" (UniqueName: \"kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j\") pod \"3212025f-ad33-4c26-b843-9417923b6d0f\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") "
Sep 15 07:29:59.115808 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.115763    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9qzt\" (UniqueName: \"kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt\") pod \"da1a6f97-c508-486b-832f-2432c140deb3\" (UID: \"da1a6f97-c508-486b-832f-2432c140deb3\") "
Sep 15 07:29:59.115864 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.115831    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume\") pod \"3212025f-ad33-4c26-b843-9417923b6d0f\" (UID: \"3212025f-ad33-4c26-b843-9417923b6d0f\") "
Sep 15 07:29:59.116319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.116283    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3212025f-ad33-4c26-b843-9417923b6d0f" (UID: "3212025f-ad33-4c26-b843-9417923b6d0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:29:59.117670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.117647    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j" (OuterVolumeSpecName: "kube-api-access-pd62j") pod "3212025f-ad33-4c26-b843-9417923b6d0f" (UID: "3212025f-ad33-4c26-b843-9417923b6d0f"). InnerVolumeSpecName "kube-api-access-pd62j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:59.117742 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.117697    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt" (OuterVolumeSpecName: "kube-api-access-b9qzt") pod "da1a6f97-c508-486b-832f-2432c140deb3" (UID: "da1a6f97-c508-486b-832f-2432c140deb3"). InnerVolumeSpecName "kube-api-access-b9qzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:29:59.204636 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.204615    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.217065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.217047    2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3212025f-ad33-4c26-b843-9417923b6d0f-extensions-socket-volume\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:59.217144 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.217067    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd62j\" (UniqueName: \"kubernetes.io/projected/3212025f-ad33-4c26-b843-9417923b6d0f-kube-api-access-pd62j\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:59.217144 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.217077    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qzt\" (UniqueName: \"kubernetes.io/projected/da1a6f97-c508-486b-832f-2432c140deb3-kube-api-access-b9qzt\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:29:59.486066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.486041    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:29:59.487234 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:29:59.487207    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b74bc4_6f69_49e3_a835_3176f2bdf262.slice/crio-dde995379aec12cafeef42340e1d1c6bf11b5d2f1bf1921da1c970a0a256c024 WatchSource:0}: Error finding container dde995379aec12cafeef42340e1d1c6bf11b5d2f1bf1921da1c970a0a256c024: Status 404 returned error can't find the container with id dde995379aec12cafeef42340e1d1c6bf11b5d2f1bf1921da1c970a0a256c024
Sep 15 07:29:59.493264 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.493237    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 07:29:59.493519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.493507    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da1a6f97-c508-486b-832f-2432c140deb3" containerName="manager"
Sep 15 07:29:59.493596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.493520    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="da1a6f97-c508-486b-832f-2432c140deb3" containerName="manager"
Sep 15 07:29:59.493596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.493579    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="da1a6f97-c508-486b-832f-2432c140deb3" containerName="manager"
Sep 15 07:29:59.495849 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.495833    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.497891 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.497868    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.499543 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.499521    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.515036 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.515012    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 07:29:59.620332 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.620307    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4v8l\" (UniqueName: \"kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.620437 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.620342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.721456 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.721429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4v8l\" (UniqueName: \"kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.721562 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.721463    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.721772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.721757    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.729353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.729332    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4v8l\" (UniqueName: \"kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l\") pod \"kuadrant-operator-controller-manager-5769f99d9f-4jvrp\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.807667 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.807645    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:29:59.931320 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.931295    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.931696 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.931304    2574 scope.go:117] "RemoveContainer" containerID="49ffb9740aeb15bec88af089fe5f5415bf20eed9180c978576a624a2979e6313"
Sep 15 07:29:59.932739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.932721    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.933301 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.933262    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.934960 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.934934    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" event={"ID":"72b74bc4-6f69-49e3-a835-3176f2bdf262","Type":"ContainerStarted","Data":"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"}
Sep 15 07:29:59.935045 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.934987    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" event={"ID":"72b74bc4-6f69-49e3-a835-3176f2bdf262","Type":"ContainerStarted","Data":"dde995379aec12cafeef42340e1d1c6bf11b5d2f1bf1921da1c970a0a256c024"}
Sep 15 07:29:59.935045 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.935025    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.935180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.935082    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:29:59.941220 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.941140    2574 scope.go:117] "RemoveContainer" containerID="6c0d6c64ac2ac548398bc258b4a65859755ac09e95c5f6f6f43d2f8f683ed909"
Sep 15 07:29:59.961609 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.961586    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:29:59.962148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:29:59.962096    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" podStartSLOduration=1.962083193 podStartE2EDuration="1.962083193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:29:59.959649983 +0000 UTC m=+688.053816244" watchObservedRunningTime="2026-09-15 07:29:59.962083193 +0000 UTC m=+688.056249456"
Sep 15 07:29:59.963807 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.963787    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.965415 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.965396    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-9c9fn\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="da1a6f97-c508-486b-832f-2432c140deb3" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-9c9fn"
Sep 15 07:29:59.966970 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:29:59.966952    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4m9g8\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4m9g8"
Sep 15 07:30:00.087638 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.087614    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 07:30:00.089692 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:00.089659    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d098681_6dc5_46de_946f_71470037c841.slice/crio-4d2a06ff66cad3f30d246ecede2d5713bbe43333517cbdeceb73a9959a82e766 WatchSource:0}: Error finding container 4d2a06ff66cad3f30d246ecede2d5713bbe43333517cbdeceb73a9959a82e766: Status 404 returned error can't find the container with id 4d2a06ff66cad3f30d246ecede2d5713bbe43333517cbdeceb73a9959a82e766
Sep 15 07:30:00.378579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.378498    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3212025f-ad33-4c26-b843-9417923b6d0f" path="/var/lib/kubelet/pods/3212025f-ad33-4c26-b843-9417923b6d0f/volumes"
Sep 15 07:30:00.378867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.378853    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da1a6f97-c508-486b-832f-2432c140deb3" path="/var/lib/kubelet/pods/da1a6f97-c508-486b-832f-2432c140deb3/volumes"
Sep 15 07:30:00.942007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.941968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" event={"ID":"8d098681-6dc5-46de-946f-71470037c841","Type":"ContainerStarted","Data":"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"}
Sep 15 07:30:00.942452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.942017    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" event={"ID":"8d098681-6dc5-46de-946f-71470037c841","Type":"ContainerStarted","Data":"4d2a06ff66cad3f30d246ecede2d5713bbe43333517cbdeceb73a9959a82e766"}
Sep 15 07:30:00.942452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.942046    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:30:00.964978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:00.964928    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" podStartSLOduration=1.964911343 podStartE2EDuration="1.964911343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:00.962627954 +0000 UTC m=+689.056794216" watchObservedRunningTime="2026-09-15 07:30:00.964911343 +0000 UTC m=+689.059077604"
Sep 15 07:30:10.945890 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:10.945860    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:30:11.951277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:11.951245    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 07:30:12.004151 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.004084    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:30:12.004392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.004349    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" podUID="72b74bc4-6f69-49e3-a835-3176f2bdf262" containerName="manager" containerID="cri-o://6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739" gracePeriod=10
Sep 15 07:30:12.261999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.261976    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:30:12.421452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.421426    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume\") pod \"72b74bc4-6f69-49e3-a835-3176f2bdf262\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") "
Sep 15 07:30:12.421588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.421525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42798\" (UniqueName: \"kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798\") pod \"72b74bc4-6f69-49e3-a835-3176f2bdf262\" (UID: \"72b74bc4-6f69-49e3-a835-3176f2bdf262\") "
Sep 15 07:30:12.421768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.421748    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "72b74bc4-6f69-49e3-a835-3176f2bdf262" (UID: "72b74bc4-6f69-49e3-a835-3176f2bdf262"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:30:12.423372 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.423345    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798" (OuterVolumeSpecName: "kube-api-access-42798") pod "72b74bc4-6f69-49e3-a835-3176f2bdf262" (UID: "72b74bc4-6f69-49e3-a835-3176f2bdf262"). InnerVolumeSpecName "kube-api-access-42798". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:30:12.522336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.522264    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42798\" (UniqueName: \"kubernetes.io/projected/72b74bc4-6f69-49e3-a835-3176f2bdf262-kube-api-access-42798\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:30:12.522336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:12.522283    2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/72b74bc4-6f69-49e3-a835-3176f2bdf262-extensions-socket-volume\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:30:13.012323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.012288    2574 generic.go:356] "Generic (PLEG): container finished" podID="72b74bc4-6f69-49e3-a835-3176f2bdf262" containerID="6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739" exitCode=0
Sep 15 07:30:13.012697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.012330    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"
Sep 15 07:30:13.012697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.012379    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" event={"ID":"72b74bc4-6f69-49e3-a835-3176f2bdf262","Type":"ContainerDied","Data":"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"}
Sep 15 07:30:13.012697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.012425    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k" event={"ID":"72b74bc4-6f69-49e3-a835-3176f2bdf262","Type":"ContainerDied","Data":"dde995379aec12cafeef42340e1d1c6bf11b5d2f1bf1921da1c970a0a256c024"}
Sep 15 07:30:13.012697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.012448    2574 scope.go:117] "RemoveContainer" containerID="6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"
Sep 15 07:30:13.021072 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.021055    2574 scope.go:117] "RemoveContainer" containerID="6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"
Sep 15 07:30:13.021374 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:30:13.021353    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739\": container with ID starting with 6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739 not found: ID does not exist" containerID="6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"
Sep 15 07:30:13.021425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.021382    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739"} err="failed to get container status \"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739\": rpc error: code = NotFound desc = could not find container \"6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739\": container with ID starting with 6c4a06a080df7e8ce1a96bb6f83cbece08a593352bb4aba72394cbf91b882739 not found: ID does not exist"
Sep 15 07:30:13.037898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.037877    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:30:13.048828 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:13.048806    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-d9s2k"]
Sep 15 07:30:14.377623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:14.377593    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b74bc4-6f69-49e3-a835-3176f2bdf262" path="/var/lib/kubelet/pods/72b74bc4-6f69-49e3-a835-3176f2bdf262/volumes"
Sep 15 07:30:28.247802 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.247722    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"]
Sep 15 07:30:28.248178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.248007    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b74bc4-6f69-49e3-a835-3176f2bdf262" containerName="manager"
Sep 15 07:30:28.248178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.248018    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b74bc4-6f69-49e3-a835-3176f2bdf262" containerName="manager"
Sep 15 07:30:28.248178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.248092    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b74bc4-6f69-49e3-a835-3176f2bdf262" containerName="manager"
Sep 15 07:30:28.251078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.251056    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.253039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.253019    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qzq9c\""
Sep 15 07:30:28.253210 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.253039    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 07:30:28.263172 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.263148    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"]
Sep 15 07:30:28.322076 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322203 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322203 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322170    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k9fj\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-kube-api-access-9k9fj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.322462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.322376    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423202    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423231    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423256    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423273    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423317    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423348    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k9fj\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-kube-api-access-9k9fj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423610    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423682    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.423929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.423900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.424092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.424075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.425597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.425577    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.425946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.425924    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.430192 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.430173    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.430431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.430414    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k9fj\" (UniqueName: \"kubernetes.io/projected/77088409-0655-4a30-a63f-1a7bc5ef1a8b-kube-api-access-9k9fj\") pod \"maas-default-gateway-openshift-default-5ccb968c66-nj4wg\" (UID: \"77088409-0655-4a30-a63f-1a7bc5ef1a8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.561269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.561214    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:28.834204 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:28.834178    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"]
Sep 15 07:30:28.835429 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:28.835400    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77088409_0655_4a30_a63f_1a7bc5ef1a8b.slice/crio-10ebfc44034038046227efe84d57614ed5df9bbc664d756a42b77c4a45741f46 WatchSource:0}: Error finding container 10ebfc44034038046227efe84d57614ed5df9bbc664d756a42b77c4a45741f46: Status 404 returned error can't find the container with id 10ebfc44034038046227efe84d57614ed5df9bbc664d756a42b77c4a45741f46
Sep 15 07:30:29.106551 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:29.106474    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" event={"ID":"77088409-0655-4a30-a63f-1a7bc5ef1a8b","Type":"ContainerStarted","Data":"10ebfc44034038046227efe84d57614ed5df9bbc664d756a42b77c4a45741f46"}
Sep 15 07:30:32.335833 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:32.335799    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:30:32.336101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:32.335866    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:30:32.336101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:32.335897    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:30:33.130585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:33.130551    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" event={"ID":"77088409-0655-4a30-a63f-1a7bc5ef1a8b","Type":"ContainerStarted","Data":"00b543eec7a9aef16f255158cb381ff0f41571e1fa066096c56b075eb5e0622b"}
Sep 15 07:30:33.153033 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:33.152982    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" podStartSLOduration=1.654919746 podStartE2EDuration="5.152965976s" totalImagesPullingTime="3.49804623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:33.152100457 +0000 UTC m=+721.246266721" watchObservedRunningTime="2026-09-15 07:30:33.152965976 +0000 UTC m=+721.247132238"
Sep 15 07:30:33.562339 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:33.562307    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:33.567216 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:33.567191    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:34.135254 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.135218    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:34.136343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.136324    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:30:34.164299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.164274    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:34.167931 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.167897    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.169801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.169779    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rkdv7\""
Sep 15 07:30:34.169801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.169779    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 07:30:34.177257 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.177227    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:34.261982 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.261952    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.262161 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.262036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8wjb\" (UniqueName: \"kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.262287 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.262259    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:34.362472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.362438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8wjb\" (UniqueName: \"kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.362652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.362498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.363146 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.363098    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.372227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.372193    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8wjb\" (UniqueName: \"kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb\") pod \"limitador-limitador-8588749977-2zc6s\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") " pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.482026 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.481997    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:34.524144 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.524094    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:30:34.528630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.528603    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.537513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.537489    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:30:34.564368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.564343    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-config-file\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.564682 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.564393    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh4dj\" (UniqueName: \"kubernetes.io/projected/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-kube-api-access-vh4dj\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.572578 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.572552    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:30:34.665563 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.665524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-config-file\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.665711 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.665630    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh4dj\" (UniqueName: \"kubernetes.io/projected/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-kube-api-access-vh4dj\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.666205 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.666186    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-config-file\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.672840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.672818    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh4dj\" (UniqueName: \"kubernetes.io/projected/7ac85e8e-0767-4b03-b51f-5f536c71d6dd-kube-api-access-vh4dj\") pod \"limitador-limitador-84994f7ffc-26rq6\" (UID: \"7ac85e8e-0767-4b03-b51f-5f536c71d6dd\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:34.754966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.754942    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:34.756717 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:34.756680    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e544646_d179_477e_943a_4f32ae30124e.slice/crio-47a05acf235c030705cf6e5831855b9d6330528d104cd8d5dc5e597d70135e3e WatchSource:0}: Error finding container 47a05acf235c030705cf6e5831855b9d6330528d104cd8d5dc5e597d70135e3e: Status 404 returned error can't find the container with id 47a05acf235c030705cf6e5831855b9d6330528d104cd8d5dc5e597d70135e3e
Sep 15 07:30:34.844057 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:34.844016    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:35.083336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.083258    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:30:35.087721 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.087697    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:30:35.089867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.089849    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rc8pz\""
Sep 15 07:30:35.092131 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.092094    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:30:35.115801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.115777    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:30:35.118362 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:35.118332    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac85e8e_0767_4b03_b51f_5f536c71d6dd.slice/crio-183ae1c783e7843c66daae306e5004513142985d0a2c97f0c3c0d00b8125316a WatchSource:0}: Error finding container 183ae1c783e7843c66daae306e5004513142985d0a2c97f0c3c0d00b8125316a: Status 404 returned error can't find the container with id 183ae1c783e7843c66daae306e5004513142985d0a2c97f0c3c0d00b8125316a
Sep 15 07:30:35.141682 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.141649    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" event={"ID":"4e544646-d179-477e-943a-4f32ae30124e","Type":"ContainerStarted","Data":"47a05acf235c030705cf6e5831855b9d6330528d104cd8d5dc5e597d70135e3e"}
Sep 15 07:30:35.142819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.142793    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6" event={"ID":"7ac85e8e-0767-4b03-b51f-5f536c71d6dd","Type":"ContainerStarted","Data":"183ae1c783e7843c66daae306e5004513142985d0a2c97f0c3c0d00b8125316a"}
Sep 15 07:30:35.169172 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.169149    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tw2b\" (UniqueName: \"kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b\") pod \"authorino-59c974bd48-pljp4\" (UID: \"47faf145-cae9-47e2-8f82-10ed13fc085e\") " pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:30:35.270165 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.270092    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tw2b\" (UniqueName: \"kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b\") pod \"authorino-59c974bd48-pljp4\" (UID: \"47faf145-cae9-47e2-8f82-10ed13fc085e\") " pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:30:35.278917 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.278896    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tw2b\" (UniqueName: \"kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b\") pod \"authorino-59c974bd48-pljp4\" (UID: \"47faf145-cae9-47e2-8f82-10ed13fc085e\") " pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:30:35.398630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.398565    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:30:35.665582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:35.665556    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:30:35.667476 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:35.667440    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47faf145_cae9_47e2_8f82_10ed13fc085e.slice/crio-888a202673543db8dd302fe968cbcd0f7fede6334fd8d4d4d07e9f5ea25dc841 WatchSource:0}: Error finding container 888a202673543db8dd302fe968cbcd0f7fede6334fd8d4d4d07e9f5ea25dc841: Status 404 returned error can't find the container with id 888a202673543db8dd302fe968cbcd0f7fede6334fd8d4d4d07e9f5ea25dc841
Sep 15 07:30:36.148599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:36.148520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pljp4" event={"ID":"47faf145-cae9-47e2-8f82-10ed13fc085e","Type":"ContainerStarted","Data":"888a202673543db8dd302fe968cbcd0f7fede6334fd8d4d4d07e9f5ea25dc841"}
Sep 15 07:30:37.156134 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.156085    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6" event={"ID":"7ac85e8e-0767-4b03-b51f-5f536c71d6dd","Type":"ContainerStarted","Data":"f01ff9f3f9ac8ba19be60fd413f4d83b8706cb83e43ec5902f88b6ef60acbaf9"}
Sep 15 07:30:37.156595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.156327    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:37.157956 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.157935    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" event={"ID":"4e544646-d179-477e-943a-4f32ae30124e","Type":"ContainerStarted","Data":"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"}
Sep 15 07:30:37.158086 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.158054    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:37.173953 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.173912    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6" podStartSLOduration=2.103505417 podStartE2EDuration="3.173891996s" totalImagesPullingTime="1.070386579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:37.1719631 +0000 UTC m=+725.266129363" watchObservedRunningTime="2026-09-15 07:30:37.173891996 +0000 UTC m=+725.268058258"
Sep 15 07:30:37.187688 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:37.187637    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" podStartSLOduration=1.757392577 podStartE2EDuration="3.187618816s" totalImagesPullingTime="1.430226239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:37.186636335 +0000 UTC m=+725.280802596" watchObservedRunningTime="2026-09-15 07:30:37.187618816 +0000 UTC m=+725.281785082"
Sep 15 07:30:38.165548 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:38.165507    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pljp4" event={"ID":"47faf145-cae9-47e2-8f82-10ed13fc085e","Type":"ContainerStarted","Data":"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"}
Sep 15 07:30:38.181220 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:38.181173    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-pljp4" podStartSLOduration=1.3969297219999999 podStartE2EDuration="3.18115922s" totalImagesPullingTime="1.784229498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:38.178801349 +0000 UTC m=+726.272967612" watchObservedRunningTime="2026-09-15 07:30:38.18115922 +0000 UTC m=+726.275325513"
Sep 15 07:30:48.166424 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.166391    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-26rq6"
Sep 15 07:30:48.166981 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.166466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:48.219165 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.219133    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:48.219322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.219283    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" podUID="4e544646-d179-477e-943a-4f32ae30124e" containerName="limitador" containerID="cri-o://05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33" gracePeriod=30
Sep 15 07:30:48.766995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.766971    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:48.887280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.887211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file\") pod \"4e544646-d179-477e-943a-4f32ae30124e\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") "
Sep 15 07:30:48.887280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.887258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8wjb\" (UniqueName: \"kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb\") pod \"4e544646-d179-477e-943a-4f32ae30124e\" (UID: \"4e544646-d179-477e-943a-4f32ae30124e\") "
Sep 15 07:30:48.887574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.887546    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file" (OuterVolumeSpecName: "config-file") pod "4e544646-d179-477e-943a-4f32ae30124e" (UID: "4e544646-d179-477e-943a-4f32ae30124e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:30:48.889279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.889261    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb" (OuterVolumeSpecName: "kube-api-access-l8wjb") pod "4e544646-d179-477e-943a-4f32ae30124e" (UID: "4e544646-d179-477e-943a-4f32ae30124e"). InnerVolumeSpecName "kube-api-access-l8wjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:30:48.987878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.987855    2574 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4e544646-d179-477e-943a-4f32ae30124e-config-file\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:30:48.987878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:48.987876    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8wjb\" (UniqueName: \"kubernetes.io/projected/4e544646-d179-477e-943a-4f32ae30124e-kube-api-access-l8wjb\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:30:49.222466 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.222439    2574 generic.go:356] "Generic (PLEG): container finished" podID="4e544646-d179-477e-943a-4f32ae30124e" containerID="05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33" exitCode=0
Sep 15 07:30:49.222833 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.222480    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s"
Sep 15 07:30:49.222833 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.222528    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" event={"ID":"4e544646-d179-477e-943a-4f32ae30124e","Type":"ContainerDied","Data":"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"}
Sep 15 07:30:49.222833 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.222601    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2zc6s" event={"ID":"4e544646-d179-477e-943a-4f32ae30124e","Type":"ContainerDied","Data":"47a05acf235c030705cf6e5831855b9d6330528d104cd8d5dc5e597d70135e3e"}
Sep 15 07:30:49.222833 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.222629    2574 scope.go:117] "RemoveContainer" containerID="05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"
Sep 15 07:30:49.231594 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.231580    2574 scope.go:117] "RemoveContainer" containerID="05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"
Sep 15 07:30:49.231831 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:30:49.231814    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33\": container with ID starting with 05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33 not found: ID does not exist" containerID="05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"
Sep 15 07:30:49.231875 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.231839    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33"} err="failed to get container status \"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33\": rpc error: code = NotFound desc = could not find container \"05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33\": container with ID starting with 05aff8c1068280eab00aa283faacf4cea5c7a92ebf44f5b035a72a1863394e33 not found: ID does not exist"
Sep 15 07:30:49.242946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.242926    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:49.246771 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:49.246750    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2zc6s"]
Sep 15 07:30:50.378099 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.378064    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e544646-d179-477e-943a-4f32ae30124e" path="/var/lib/kubelet/pods/4e544646-d179-477e-943a-4f32ae30124e/volumes"
Sep 15 07:30:50.589541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.589510    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"]
Sep 15 07:30:50.589847 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.589835    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e544646-d179-477e-943a-4f32ae30124e" containerName="limitador"
Sep 15 07:30:50.589893 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.589850    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e544646-d179-477e-943a-4f32ae30124e" containerName="limitador"
Sep 15 07:30:50.589927 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.589900    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e544646-d179-477e-943a-4f32ae30124e" containerName="limitador"
Sep 15 07:30:50.594307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.594286    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.596337 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.596316    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 15 07:30:50.596469 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.596382    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 07:30:50.596910 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.596886    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wbwp6\""
Sep 15 07:30:50.597006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.596887    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 07:30:50.601082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.601058    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"]
Sep 15 07:30:50.703184 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.703156    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj578\" (UniqueName: \"kubernetes.io/projected/65eff245-4884-478f-bc0c-994bc46f6791-kube-api-access-wj578\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.703353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.703217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65eff245-4884-478f-bc0c-994bc46f6791-data\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.804140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.804074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj578\" (UniqueName: \"kubernetes.io/projected/65eff245-4884-478f-bc0c-994bc46f6791-kube-api-access-wj578\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.804336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.804185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65eff245-4884-478f-bc0c-994bc46f6791-data\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.804568 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.804545    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/65eff245-4884-478f-bc0c-994bc46f6791-data\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.812105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.812072    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj578\" (UniqueName: \"kubernetes.io/projected/65eff245-4884-478f-bc0c-994bc46f6791-kube-api-access-wj578\") pod \"postgres-69fb4594cf-rjvd6\" (UID: \"65eff245-4884-478f-bc0c-994bc46f6791\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:50.906935 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:50.906905    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:51.179441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:51.179415    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"]
Sep 15 07:30:51.181584 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:30:51.181552    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eff245_4884_478f_bc0c_994bc46f6791.slice/crio-bf9e920397665056db038302164c028586149b5d714e26ae8090227eded557ce WatchSource:0}: Error finding container bf9e920397665056db038302164c028586149b5d714e26ae8090227eded557ce: Status 404 returned error can't find the container with id bf9e920397665056db038302164c028586149b5d714e26ae8090227eded557ce
Sep 15 07:30:51.234350 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:51.234269    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6" event={"ID":"65eff245-4884-478f-bc0c-994bc46f6791","Type":"ContainerStarted","Data":"bf9e920397665056db038302164c028586149b5d714e26ae8090227eded557ce"}
Sep 15 07:30:56.266189 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:56.266155    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6" event={"ID":"65eff245-4884-478f-bc0c-994bc46f6791","Type":"ContainerStarted","Data":"11b29feab78fe4a951ab37ec39b5df420029ca78dc5f1fdbdf23f85217b098fb"}
Sep 15 07:30:56.266595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:56.266334    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:30:56.280334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:30:56.280266    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6" podStartSLOduration=1.321013121 podStartE2EDuration="6.28024675s" totalImagesPullingTime="4.959233629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:30:56.28022106 +0000 UTC m=+744.374387323" watchObservedRunningTime="2026-09-15 07:30:56.28024675 +0000 UTC m=+744.374413037"
Sep 15 07:31:02.300222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:02.300177    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-rjvd6"
Sep 15 07:31:05.047739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.047701    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:31:05.048107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.047882    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-pljp4" podUID="47faf145-cae9-47e2-8f82-10ed13fc085e" containerName="authorino" containerID="cri-o://d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881" gracePeriod=30
Sep 15 07:31:05.300056 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.299999    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:31:05.314278 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.314247    2574 generic.go:356] "Generic (PLEG): container finished" podID="47faf145-cae9-47e2-8f82-10ed13fc085e" containerID="d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881" exitCode=0
Sep 15 07:31:05.314376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.314291    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-pljp4"
Sep 15 07:31:05.314376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.314309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pljp4" event={"ID":"47faf145-cae9-47e2-8f82-10ed13fc085e","Type":"ContainerDied","Data":"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"}
Sep 15 07:31:05.314376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.314355    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-pljp4" event={"ID":"47faf145-cae9-47e2-8f82-10ed13fc085e","Type":"ContainerDied","Data":"888a202673543db8dd302fe968cbcd0f7fede6334fd8d4d4d07e9f5ea25dc841"}
Sep 15 07:31:05.314499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.314377    2574 scope.go:117] "RemoveContainer" containerID="d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"
Sep 15 07:31:05.324107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.323953    2574 scope.go:117] "RemoveContainer" containerID="d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"
Sep 15 07:31:05.324377 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:31:05.324355    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881\": container with ID starting with d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881 not found: ID does not exist" containerID="d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"
Sep 15 07:31:05.324428 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.324388    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881"} err="failed to get container status \"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881\": rpc error: code = NotFound desc = could not find container \"d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881\": container with ID starting with d4b66e0ab751c76e9a24e18e9d3f74cf7fac28042535d2f74f86355f98d68881 not found: ID does not exist"
Sep 15 07:31:05.431662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.431632    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tw2b\" (UniqueName: \"kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b\") pod \"47faf145-cae9-47e2-8f82-10ed13fc085e\" (UID: \"47faf145-cae9-47e2-8f82-10ed13fc085e\") "
Sep 15 07:31:05.433768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.433747    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b" (OuterVolumeSpecName: "kube-api-access-2tw2b") pod "47faf145-cae9-47e2-8f82-10ed13fc085e" (UID: "47faf145-cae9-47e2-8f82-10ed13fc085e"). InnerVolumeSpecName "kube-api-access-2tw2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:31:05.532427 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.532393    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tw2b\" (UniqueName: \"kubernetes.io/projected/47faf145-cae9-47e2-8f82-10ed13fc085e-kube-api-access-2tw2b\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:31:05.651074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.651048    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:31:05.670190 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:05.670162    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-pljp4"]
Sep 15 07:31:06.377479 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:06.377438    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47faf145-cae9-47e2-8f82-10ed13fc085e" path="/var/lib/kubelet/pods/47faf145-cae9-47e2-8f82-10ed13fc085e/volumes"
Sep 15 07:31:09.904074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.904040    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:31:09.904547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.904432    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47faf145-cae9-47e2-8f82-10ed13fc085e" containerName="authorino"
Sep 15 07:31:09.904547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.904445    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="47faf145-cae9-47e2-8f82-10ed13fc085e" containerName="authorino"
Sep 15 07:31:09.904547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.904497    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="47faf145-cae9-47e2-8f82-10ed13fc085e" containerName="authorino"
Sep 15 07:31:09.919587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.919553    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:31:09.919747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.919634    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:09.921774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.921751    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vw7xh\""
Sep 15 07:31:09.922476 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.922455    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 07:31:09.922599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.922492    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 07:31:09.922599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:09.922531    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 07:31:10.071182 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.071150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.071354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.071207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xljvq\" (UniqueName: \"kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.071354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.071324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.172406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.172380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.172559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.172419    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.172559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.172467    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xljvq\" (UniqueName: \"kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.174858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.174833    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.174972 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.174868    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.179869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.179850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xljvq\" (UniqueName: \"kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq\") pod \"maas-controller-656b58fbfb-8wlgp\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") " pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.230709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.230669    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:10.499957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:10.499903    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:31:10.502606 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:31:10.502581    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa53b6fb_d972_4b11_9179_ce79cf082c80.slice/crio-4904cf6c1d743ebfdcb05e6563b6556820bcd61db66e274e5f857b71f3477c18 WatchSource:0}: Error finding container 4904cf6c1d743ebfdcb05e6563b6556820bcd61db66e274e5f857b71f3477c18: Status 404 returned error can't find the container with id 4904cf6c1d743ebfdcb05e6563b6556820bcd61db66e274e5f857b71f3477c18
Sep 15 07:31:11.348861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:11.348825    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" event={"ID":"aa53b6fb-d972-4b11-9179-ce79cf082c80","Type":"ContainerStarted","Data":"4904cf6c1d743ebfdcb05e6563b6556820bcd61db66e274e5f857b71f3477c18"}
Sep 15 07:31:13.360987 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:13.360951    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" event={"ID":"aa53b6fb-d972-4b11-9179-ce79cf082c80","Type":"ContainerStarted","Data":"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"}
Sep 15 07:31:13.361399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:13.361078    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:13.377377 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:13.377092    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" podStartSLOduration=1.735061642 podStartE2EDuration="4.377075159s" totalImagesPullingTime="2.642013517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:31:13.376677136 +0000 UTC m=+761.470843398" watchObservedRunningTime="2026-09-15 07:31:13.377075159 +0000 UTC m=+761.471241422"
Sep 15 07:31:24.369487 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:24.369451    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:31:46.402707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.402668    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 07:31:46.409095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.409066    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.411085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.411064    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 07:31:46.411273 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.411250    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 07:31:46.411346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.411257    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xf69m\""
Sep 15 07:31:46.415978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.415956    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 07:31:46.482511 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.482477    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.482792 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.482529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.482792 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.482552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.482792 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.482642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxn59\" (UniqueName: \"kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.583460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.583428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxn59\" (UniqueName: \"kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.583628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.583472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.583628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.583498    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.583628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.583516    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.584149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.584103    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.586081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.586058    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.586081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.586069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.591494 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.591471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxn59\" (UniqueName: \"kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59\") pod \"maas-api-66bb9979b8-2xntc\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") " pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:46.721473 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:46.721448    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:47.003018 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:47.002992    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 07:31:47.004965 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:31:47.004928    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f504830_d566_40d4_b037_4823b074978e.slice/crio-ac040b95d1f8eca6108e8f50417dd8ece1b6fb3b7a98ef0db7f5d29735d2f4a9 WatchSource:0}: Error finding container ac040b95d1f8eca6108e8f50417dd8ece1b6fb3b7a98ef0db7f5d29735d2f4a9: Status 404 returned error can't find the container with id ac040b95d1f8eca6108e8f50417dd8ece1b6fb3b7a98ef0db7f5d29735d2f4a9
Sep 15 07:31:47.083775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:47.083747    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:31:47.528317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:47.528276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" event={"ID":"9f504830-d566-40d4-b037-4823b074978e","Type":"ContainerStarted","Data":"ac040b95d1f8eca6108e8f50417dd8ece1b6fb3b7a98ef0db7f5d29735d2f4a9"}
Sep 15 07:31:49.541053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:49.541015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" event={"ID":"9f504830-d566-40d4-b037-4823b074978e","Type":"ContainerStarted","Data":"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"}
Sep 15 07:31:49.541446 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:49.541111    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:49.559730 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:49.559670    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" podStartSLOduration=1.657299107 podStartE2EDuration="3.559648417s" totalImagesPullingTime="1.90234931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:31:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:31:49.556075374 +0000 UTC m=+797.650241651" watchObservedRunningTime="2026-09-15 07:31:49.559648417 +0000 UTC m=+797.653814680"
Sep 15 07:31:55.550612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:55.550538    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 07:31:56.585525 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.585438    2574 generic.go:356] "Generic (PLEG): container finished" podID="77088409-0655-4a30-a63f-1a7bc5ef1a8b" containerID="00b543eec7a9aef16f255158cb381ff0f41571e1fa066096c56b075eb5e0622b" exitCode=0
Sep 15 07:31:56.586023 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.585515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" event={"ID":"77088409-0655-4a30-a63f-1a7bc5ef1a8b","Type":"ContainerDied","Data":"00b543eec7a9aef16f255158cb381ff0f41571e1fa066096c56b075eb5e0622b"}
Sep 15 07:31:56.586023 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.585995    2574 scope.go:117] "RemoveContainer" containerID="00b543eec7a9aef16f255158cb381ff0f41571e1fa066096c56b075eb5e0622b"
Sep 15 07:31:56.587016 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.586988    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:31:56.587152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.587070    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:31:56.587152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:56.587145    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:31:57.593107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:57.593077    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" event={"ID":"77088409-0655-4a30-a63f-1a7bc5ef1a8b","Type":"ContainerStarted","Data":"e75436f9e977a3746508c08461bf4b19ccf7702ddb0a4df36b6d6ea3295f0612"}
Sep 15 07:31:58.561813 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:58.561768    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:31:58.562222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:58.562192    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nj4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 07:31:58.562430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:58.562405    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" podUID="77088409-0655-4a30-a63f-1a7bc5ef1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 07:31:59.562460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:59.561983    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nj4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 07:31:59.562460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:31:59.562062    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" podUID="77088409-0655-4a30-a63f-1a7bc5ef1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 07:32:00.562376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:00.561936    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nj4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 07:32:00.562376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:00.561993    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" podUID="77088409-0655-4a30-a63f-1a7bc5ef1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 07:32:01.562334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:01.562296    2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-nj4wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 15 07:32:01.562617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:01.562588    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg" podUID="77088409-0655-4a30-a63f-1a7bc5ef1a8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 15 07:32:02.566223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:02.566192    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:32:02.566642 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:02.566538    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:32:02.567329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:02.567310    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-nj4wg"
Sep 15 07:32:12.011644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:12.011607    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:32:19.606060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:19.606014    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:32:42.902725 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:32:42.902691    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:33:01.807846 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:33:01.807802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:33:14.504690 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:33:14.504653    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:33:27.904899 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:33:27.904823    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:33:52.308840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:33:52.308803    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:34:55.049461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.049375    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75bcc56485-7tvtl"]
Sep 15 07:34:55.052879 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.052858    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.055440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.055416    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 15 07:34:55.055557 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.055438    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rc8pz\""
Sep 15 07:34:55.059677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.059652    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75bcc56485-7tvtl"]
Sep 15 07:34:55.096856 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.096823    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n76sf\" (UniqueName: \"kubernetes.io/projected/e886b953-6c7c-4127-8569-cf75a55ecba8-kube-api-access-n76sf\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.096996 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.096886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e886b953-6c7c-4127-8569-cf75a55ecba8-tls-cert\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.198321 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.198277    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n76sf\" (UniqueName: \"kubernetes.io/projected/e886b953-6c7c-4127-8569-cf75a55ecba8-kube-api-access-n76sf\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.198518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.198370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e886b953-6c7c-4127-8569-cf75a55ecba8-tls-cert\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.200819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.200791    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e886b953-6c7c-4127-8569-cf75a55ecba8-tls-cert\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.206410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.206383    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n76sf\" (UniqueName: \"kubernetes.io/projected/e886b953-6c7c-4127-8569-cf75a55ecba8-kube-api-access-n76sf\") pod \"authorino-75bcc56485-7tvtl\" (UID: \"e886b953-6c7c-4127-8569-cf75a55ecba8\") " pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.363780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.363700    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75bcc56485-7tvtl"
Sep 15 07:34:55.633892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.633867    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75bcc56485-7tvtl"]
Sep 15 07:34:55.635490 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:34:55.635465    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode886b953_6c7c_4127_8569_cf75a55ecba8.slice/crio-537542d4877603c0d7703d76b493f55af761d932ce45aad79e861ddefb3e5ce3 WatchSource:0}: Error finding container 537542d4877603c0d7703d76b493f55af761d932ce45aad79e861ddefb3e5ce3: Status 404 returned error can't find the container with id 537542d4877603c0d7703d76b493f55af761d932ce45aad79e861ddefb3e5ce3
Sep 15 07:34:55.636734 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:55.636713    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:34:56.511468 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:56.511417    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bcc56485-7tvtl" event={"ID":"e886b953-6c7c-4127-8569-cf75a55ecba8","Type":"ContainerStarted","Data":"e2612683160d4e453d0a2ff3778162cabf61545573c7f45eda04a3d977401d30"}
Sep 15 07:34:56.511827 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:56.511486    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75bcc56485-7tvtl" event={"ID":"e886b953-6c7c-4127-8569-cf75a55ecba8","Type":"ContainerStarted","Data":"537542d4877603c0d7703d76b493f55af761d932ce45aad79e861ddefb3e5ce3"}
Sep 15 07:34:56.530229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:34:56.530103    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75bcc56485-7tvtl" podStartSLOduration=0.902656646 podStartE2EDuration="1.530082987s" totalImagesPullingTime="627.426341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:34:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:34:56.528019743 +0000 UTC m=+984.622186008" watchObservedRunningTime="2026-09-15 07:34:56.530082987 +0000 UTC m=+984.624249250"
Sep 15 07:35:14.280398 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.280368    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:35:14.283820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.283804    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.285853 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.285826    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-f56165-openshift-default-dockercfg-2h8w8\""
Sep 15 07:35:14.295501 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.295481    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:35:14.342978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.342950    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:35:14.346573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.346550    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.349650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.349617    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ee0096-openshift-default-dockercfg-9tfhn\""
Sep 15 07:35:14.355480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355458    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355489    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps82g\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355618    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355712    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355785    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.355873 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.355822    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.369528 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.369506    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:35:14.456478 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456445    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.456478 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456481    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456497    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.456713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456688    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456738    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456775    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.456921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456909    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456922    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456961    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.456997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457053    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457152    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457139    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps82g\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.457623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457237    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dqvq\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.457623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457342    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.458002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.457978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.459482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.459454    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.459740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.459722    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.470217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.470195    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps82g\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.470513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.470492    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token\") pod \"e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") " pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.558291 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558212    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558291 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558262    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558295    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558338    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558390    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dqvq\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558691 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558691 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558644    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558691 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558657    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558838 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558838 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558710    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.558838 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.558739    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.559168 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.559139    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.559303 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.559259    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.560974 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.560942    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.561179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.561162    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.565741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.565717    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.565825 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.565726    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dqvq\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq\") pod \"e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") " pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.593687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.593664    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:14.658582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.658554    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:14.877291 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.874195    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:35:14.879389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.879355    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:14.879500 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.879434    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:14.879564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.879515    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:14.934916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.934894    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:35:14.937322 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:14.937299    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2467ee47_5d7e_4a95_a633_e3402abe1289.slice/crio-b9beac6c846ea00d31e840e23931656c6d668f1b036d0c6dcdafb056bbd1693f WatchSource:0}: Error finding container b9beac6c846ea00d31e840e23931656c6d668f1b036d0c6dcdafb056bbd1693f: Status 404 returned error can't find the container with id b9beac6c846ea00d31e840e23931656c6d668f1b036d0c6dcdafb056bbd1693f
Sep 15 07:35:14.939355 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.939326    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:14.939431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.939383    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:14.939431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:14.939410    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:15.609550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.609513    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerStarted","Data":"35ca8dfb8146b8fdc977efb966b2fd77e4f1a9d79b637ccd86f1e06c82e8852b"}
Sep 15 07:35:15.610005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.609572    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerStarted","Data":"c4b666eddb2a3c70d52414d306c088680949205faccf862468538742e1e4ddbc"}
Sep 15 07:35:15.611003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.610980    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerStarted","Data":"db1f3ef91cc70e6198d040932ae2679b041f1acff5f25d8fd79e50229ecb9f9d"}
Sep 15 07:35:15.611107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.611016    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerStarted","Data":"b9beac6c846ea00d31e840e23931656c6d668f1b036d0c6dcdafb056bbd1693f"}
Sep 15 07:35:15.633455 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.633403    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podStartSLOduration=1.633384429 podStartE2EDuration="1.633384429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:15.631991766 +0000 UTC m=+1003.726158029" watchObservedRunningTime="2026-09-15 07:35:15.633384429 +0000 UTC m=+1003.727550692"
Sep 15 07:35:15.655074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.655023    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podStartSLOduration=1.6550091820000001 podStartE2EDuration="1.655009182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:15.652103318 +0000 UTC m=+1003.746269579" watchObservedRunningTime="2026-09-15 07:35:15.655009182 +0000 UTC m=+1003.749175482"
Sep 15 07:35:15.659664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.659640    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:15.664297 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.664277    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:15.944020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.943985    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7669-1b5e22"]
Sep 15 07:35:15.947387 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.947360    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:15.949612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.949588    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8zjf\""
Sep 15 07:35:15.952042 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:15.952012    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7669-1b5e22"]
Sep 15 07:35:16.073664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.073630    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr8kx\" (UniqueName: \"kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx\") pod \"test-curl-7669-1b5e22\" (UID: \"6e99dab3-d133-410d-a9cd-d72f8f8dfd73\") " pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:16.174169 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.174132    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr8kx\" (UniqueName: \"kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx\") pod \"test-curl-7669-1b5e22\" (UID: \"6e99dab3-d133-410d-a9cd-d72f8f8dfd73\") " pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:16.181720 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.181698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr8kx\" (UniqueName: \"kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx\") pod \"test-curl-7669-1b5e22\" (UID: \"6e99dab3-d133-410d-a9cd-d72f8f8dfd73\") " pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:16.260647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.260568    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:16.529141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.529103    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7669-1b5e22"]
Sep 15 07:35:16.530921 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:16.530896    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e99dab3_d133_410d_a9cd_d72f8f8dfd73.slice/crio-7589c2655f49a48c190b1e74fb8990d24589c2b896c434113eb6b19c912a3d56 WatchSource:0}: Error finding container 7589c2655f49a48c190b1e74fb8990d24589c2b896c434113eb6b19c912a3d56: Status 404 returned error can't find the container with id 7589c2655f49a48c190b1e74fb8990d24589c2b896c434113eb6b19c912a3d56
Sep 15 07:35:16.594013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.593987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:16.598929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.598907    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:16.617301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.617266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7669-1b5e22" event={"ID":"6e99dab3-d133-410d-a9cd-d72f8f8dfd73","Type":"ContainerStarted","Data":"7589c2655f49a48c190b1e74fb8990d24589c2b896c434113eb6b19c912a3d56"}
Sep 15 07:35:16.617810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.617684    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:16.617810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.617732    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:16.618514 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.618493    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:16.618719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:16.618703    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:18.167824 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.167794    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:35:18.631412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.631312    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7669-1b5e22" event={"ID":"6e99dab3-d133-410d-a9cd-d72f8f8dfd73","Type":"ContainerStarted","Data":"ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5"}
Sep 15 07:35:18.646686 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.646636    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7669-1b5e22" podStartSLOduration=2.446379676 podStartE2EDuration="3.646620557s" totalImagesPullingTime="1.200240881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:18.644185908 +0000 UTC m=+1006.738352184" watchObservedRunningTime="2026-09-15 07:35:18.646620557 +0000 UTC m=+1006.740786818"
Sep 15 07:35:18.900669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.900596    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:35:18.904039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.904017    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:18.906212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.906186    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-2735c5-serving-cert\""
Sep 15 07:35:18.914516 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.914489    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:35:18.998518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.998492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44ds4\" (UniqueName: \"kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:18.998648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.998532    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:18.998648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.998556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:18.998648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:18.998627    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.099379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.099341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44ds4\" (UniqueName: \"kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.099559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.099391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.099559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.099421    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.099559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.099461    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.099736 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:19.099611    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-serving-cert: secret "maas-api-e2e-worker-w0-2735c5-serving-cert" not found
Sep 15 07:35:19.099736 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:19.099673    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls podName:c42acb8e-dc2b-46f9-9d01-be252b9209b8 nodeName:}" failed. No retries permitted until 2026-09-15 07:35:19.59965212 +0000 UTC m=+1007.693818361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls") pod "maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" (UID: "c42acb8e-dc2b-46f9-9d01-be252b9209b8") : secret "maas-api-e2e-worker-w0-2735c5-serving-cert" not found
Sep 15 07:35:19.100209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.100176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.102387 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.102357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.117606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.117575    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44ds4\" (UniqueName: \"kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.225698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.225670    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7669-1b5e22"]
Sep 15 07:35:19.235272 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.235248    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7669-1b5e22"]
Sep 15 07:35:19.351556 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.351526    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7669-95c529"]
Sep 15 07:35:19.352051 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.352030    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" containerName="test-curl-7669-1b5e22"
Sep 15 07:35:19.352111 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.352056    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" containerName="test-curl-7669-1b5e22"
Sep 15 07:35:19.352205 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.352191    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" containerName="test-curl-7669-1b5e22"
Sep 15 07:35:19.355608 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.355591    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:19.361724 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.361698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7669-95c529"]
Sep 15 07:35:19.503832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.503743    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gwq6\" (UniqueName: \"kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6\") pod \"test-curl-7669-95c529\" (UID: \"2f250fa1-038a-4b53-a2ef-70b410a6135a\") " pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:19.605079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.605040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gwq6\" (UniqueName: \"kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6\") pod \"test-curl-7669-95c529\" (UID: \"2f250fa1-038a-4b53-a2ef-70b410a6135a\") " pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:19.605297 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.605158    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.607718 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.607689    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") pod \"maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.614464 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.614428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gwq6\" (UniqueName: \"kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6\") pod \"test-curl-7669-95c529\" (UID: \"2f250fa1-038a-4b53-a2ef-70b410a6135a\") " pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:19.637215 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:19.637192    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7669-1b5e22" container="test-curl-7669-1b5e22"
Sep 15 07:35:19.666713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.666682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:19.816389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.816299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:19.946605 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:19.946581    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7669-95c529"]
Sep 15 07:35:19.948370 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:19.948340    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f250fa1_038a_4b53_a2ef_70b410a6135a.slice/crio-3d9bb23d904e7efae0b79d2aaab193d9ed3569dc0357adc6572ebe5b243cfcf2 WatchSource:0}: Error finding container 3d9bb23d904e7efae0b79d2aaab193d9ed3569dc0357adc6572ebe5b243cfcf2: Status 404 returned error can't find the container with id 3d9bb23d904e7efae0b79d2aaab193d9ed3569dc0357adc6572ebe5b243cfcf2
Sep 15 07:35:20.094866 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.094842    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:35:20.096414 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:20.096386    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42acb8e_dc2b_46f9_9d01_be252b9209b8.slice/crio-612949a56ad8f87740b5f1068b6fab386b7c751196247de3a3b53b2eb3c980de WatchSource:0}: Error finding container 612949a56ad8f87740b5f1068b6fab386b7c751196247de3a3b53b2eb3c980de: Status 404 returned error can't find the container with id 612949a56ad8f87740b5f1068b6fab386b7c751196247de3a3b53b2eb3c980de
Sep 15 07:35:20.117062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.117025    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:35:20.121005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.120985    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.123013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.122990    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f56165-serving-cert\""
Sep 15 07:35:20.132980 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.132948    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:35:20.211636 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.211598    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.211762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.211658    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sckd\" (UniqueName: \"kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.211799 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.211768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.211858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.211844    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.312925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.312885    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.313340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.312941    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.313340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.312964    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sckd\" (UniqueName: \"kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.313340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.313010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.313340 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:20.313186    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-serving-cert: secret "maas-api-e2e-worker-w4-f56165-serving-cert" not found
Sep 15 07:35:20.313340 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:20.313246    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls podName:e5172b04-3747-4acb-b1dd-3036a519ec9b nodeName:}" failed. No retries permitted until 2026-09-15 07:35:20.813226913 +0000 UTC m=+1008.907393161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls") pod "maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" (UID: "e5172b04-3747-4acb-b1dd-3036a519ec9b") : secret "maas-api-e2e-worker-w4-f56165-serving-cert" not found
Sep 15 07:35:20.313728 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.313707    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.315490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.315471    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.323815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.323793    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sckd\" (UniqueName: \"kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.643556 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.643522    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" event={"ID":"c42acb8e-dc2b-46f9-9d01-be252b9209b8","Type":"ContainerStarted","Data":"612949a56ad8f87740b5f1068b6fab386b7c751196247de3a3b53b2eb3c980de"}
Sep 15 07:35:20.645218 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.645179    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7669-95c529" event={"ID":"2f250fa1-038a-4b53-a2ef-70b410a6135a","Type":"ContainerStarted","Data":"52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308"}
Sep 15 07:35:20.645218 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.645200    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7669-1b5e22" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" containerName="test-curl-7669-1b5e22" containerID="cri-o://ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5" gracePeriod=2
Sep 15 07:35:20.645218 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.645220    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7669-95c529" event={"ID":"2f250fa1-038a-4b53-a2ef-70b410a6135a","Type":"ContainerStarted","Data":"3d9bb23d904e7efae0b79d2aaab193d9ed3569dc0357adc6572ebe5b243cfcf2"}
Sep 15 07:35:20.667214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.667156    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7669-95c529" podStartSLOduration=1.6671021160000001 podStartE2EDuration="1.667102116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:20.663445684 +0000 UTC m=+1008.757611947" watchObservedRunningTime="2026-09-15 07:35:20.667102116 +0000 UTC m=+1008.761268379"
Sep 15 07:35:20.817638 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.817570    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:20.820014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:20.819992    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:21.019145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.019091    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:35:21.022545 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.022525    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.024995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.024973    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ee0096-serving-cert\""
Sep 15 07:35:21.030454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.030433    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:35:21.032182 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.032132    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:21.120080 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.120036    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnppf\" (UniqueName: \"kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.120309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.120285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.120427 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.120325    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.120508 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.120442    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.221629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.221596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.221780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.221643    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnppf\" (UniqueName: \"kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.221780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.221683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.221780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.221703    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.222385 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.222362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.224141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.224094    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.224353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.224331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.229581 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.229558    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnppf\" (UniqueName: \"kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf\") pod \"maas-api-e2e-inf-a-ee0096-7557794bc7-knl54\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.307854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.307786    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7669-95c529"]
Sep 15 07:35:21.312053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.312017    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:35:21.313464 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:21.313438    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5172b04_3747_4acb_b1dd_3036a519ec9b.slice/crio-1864eab42477585a83e3855e7391c7be4728ad119a3e63b46234ad9b2960bc41 WatchSource:0}: Error finding container 1864eab42477585a83e3855e7391c7be4728ad119a3e63b46234ad9b2960bc41: Status 404 returned error can't find the container with id 1864eab42477585a83e3855e7391c7be4728ad119a3e63b46234ad9b2960bc41
Sep 15 07:35:21.316188 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.316168    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7669-95c529"]
Sep 15 07:35:21.335954 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.335933    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:21.653726 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.653634    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" event={"ID":"e5172b04-3747-4acb-b1dd-3036a519ec9b","Type":"ContainerStarted","Data":"1864eab42477585a83e3855e7391c7be4728ad119a3e63b46234ad9b2960bc41"}
Sep 15 07:35:21.654907 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.654881    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:35:21.655435 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.655409    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" event={"ID":"c42acb8e-dc2b-46f9-9d01-be252b9209b8","Type":"ContainerStarted","Data":"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"}
Sep 15 07:35:21.655574 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:21.655561    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7669-95c529" container="test-curl-7669-95c529"
Sep 15 07:35:21.657675 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:21.657651    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddded2457_2a7c_47f5_b50a_17f672a35c8e.slice/crio-aa5013961e9d7fdea45b7fbd58bdb756643b2f1f00627e00ec7dcc3a7045fb0c WatchSource:0}: Error finding container aa5013961e9d7fdea45b7fbd58bdb756643b2f1f00627e00ec7dcc3a7045fb0c: Status 404 returned error can't find the container with id aa5013961e9d7fdea45b7fbd58bdb756643b2f1f00627e00ec7dcc3a7045fb0c
Sep 15 07:35:21.685788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:21.685743    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" podStartSLOduration=2.969878461 podStartE2EDuration="3.685728897s" totalImagesPullingTime="715.850436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:21.6838782 +0000 UTC m=+1009.778044461" watchObservedRunningTime="2026-09-15 07:35:21.685728897 +0000 UTC m=+1009.779895158"
Sep 15 07:35:22.664671 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.664642    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" event={"ID":"dded2457-2a7c-47f5-b50a-17f672a35c8e","Type":"ContainerStarted","Data":"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"}
Sep 15 07:35:22.665002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.664714    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" event={"ID":"dded2457-2a7c-47f5-b50a-17f672a35c8e","Type":"ContainerStarted","Data":"aa5013961e9d7fdea45b7fbd58bdb756643b2f1f00627e00ec7dcc3a7045fb0c"}
Sep 15 07:35:22.665002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.664763    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:22.666461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.666433    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" event={"ID":"e5172b04-3747-4acb-b1dd-3036a519ec9b","Type":"ContainerStarted","Data":"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"}
Sep 15 07:35:22.666565 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.666484    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:22.666624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.666603    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:22.666715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.666695    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7669-95c529" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" containerName="test-curl-7669-95c529" containerID="cri-o://52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308" gracePeriod=2
Sep 15 07:35:22.686451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.686412    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" podStartSLOduration=2.147211007 podStartE2EDuration="2.686398261s" totalImagesPullingTime="539.187254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:22.684736618 +0000 UTC m=+1010.778902883" watchObservedRunningTime="2026-09-15 07:35:22.686398261 +0000 UTC m=+1010.780564514"
Sep 15 07:35:22.705814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.705770    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" podStartSLOduration=2.229759059 podStartE2EDuration="2.705756482s" totalImagesPullingTime="475.997423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:22.704408483 +0000 UTC m=+1010.798574745" watchObservedRunningTime="2026-09-15 07:35:22.705756482 +0000 UTC m=+1010.799922780"
Sep 15 07:35:22.830021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.829994    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:22.938540 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.938507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rr8kx\" (UniqueName: \"kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx\") pod \"6e99dab3-d133-410d-a9cd-d72f8f8dfd73\" (UID: \"6e99dab3-d133-410d-a9cd-d72f8f8dfd73\") "
Sep 15 07:35:22.940842 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:22.940810    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx" (OuterVolumeSpecName: "kube-api-access-rr8kx") pod "6e99dab3-d133-410d-a9cd-d72f8f8dfd73" (UID: "6e99dab3-d133-410d-a9cd-d72f8f8dfd73"). InnerVolumeSpecName "kube-api-access-rr8kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:35:23.039618 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.039588    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr8kx\" (UniqueName: \"kubernetes.io/projected/6e99dab3-d133-410d-a9cd-d72f8f8dfd73-kube-api-access-rr8kx\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:35:23.673786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.673751    2574 generic.go:356] "Generic (PLEG): container finished" podID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" containerID="ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5" exitCode=137
Sep 15 07:35:23.674207 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.673809    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:23.674207 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.673819    2574 scope.go:117] "RemoveContainer" containerID="ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5"
Sep 15 07:35:23.682789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.682775    2574 scope.go:117] "RemoveContainer" containerID="ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5"
Sep 15 07:35:23.683024 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:23.683007    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5\": container with ID starting with ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5 not found: ID does not exist" containerID="ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5"
Sep 15 07:35:23.683069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.683031    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5"} err="failed to get container status \"ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5\": rpc error: code = NotFound desc = could not find container \"ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5\": container with ID starting with ec1fa54221944e77b202c5e2f7379bcbae444c8b1feb7616dc10abd42a8fe9d5 not found: ID does not exist"
Sep 15 07:35:23.683825 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:23.683806    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7669-1b5e22\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:23.879903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.879868    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:35:23.880413 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.880394    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" containerName="test-curl-7669-95c529"
Sep 15 07:35:23.880413 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.880413    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" containerName="test-curl-7669-95c529"
Sep 15 07:35:23.880578 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.880505    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" containerName="test-curl-7669-95c529"
Sep 15 07:35:23.885094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.885071    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:23.886931 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:23.886883    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7669-1b5e22\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" pod="opendatahub/test-curl-7669-1b5e22"
Sep 15 07:35:23.887420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.887394    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e8faa2dc-openshift-default-dockercfg-2f644\""
Sep 15 07:35:23.897504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:23.897482    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:35:24.050098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.049996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050050    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050093    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050141    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050206    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nsjq\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050322    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.050399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.050364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151374    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151539    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151559    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.151640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151604    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nsjq\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.152013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151822    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.152013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151924    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.152013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.151983    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.152187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.152143    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.152615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.152591    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.154319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.154298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.154556 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.154537    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.164646 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.164626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.164763 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.164693    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nsjq\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq\") pod \"e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") " pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.205458 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.205430    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:24.380257 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.380184    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e99dab3-d133-410d-a9cd-d72f8f8dfd73" path="/var/lib/kubelet/pods/6e99dab3-d133-410d-a9cd-d72f8f8dfd73/volumes"
Sep 15 07:35:24.485183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.485157    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:35:24.486492 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:24.486467    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9960ee4_ff91_4514_bb4a_8a32a90646ab.slice/crio-e8f9b898ff2683556e7204c71ca7201d16a5cb0836494e19a2e383f4e44ed464 WatchSource:0}: Error finding container e8f9b898ff2683556e7204c71ca7201d16a5cb0836494e19a2e383f4e44ed464: Status 404 returned error can't find the container with id e8f9b898ff2683556e7204c71ca7201d16a5cb0836494e19a2e383f4e44ed464
Sep 15 07:35:24.488804 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.488762    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:24.488892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.488852    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:24.488939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.488923    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:24.683761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.683726    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerStarted","Data":"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"}
Sep 15 07:35:24.684064 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.683788    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerStarted","Data":"e8f9b898ff2683556e7204c71ca7201d16a5cb0836494e19a2e383f4e44ed464"}
Sep 15 07:35:24.732917 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.732867    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podStartSLOduration=1.732844783 podStartE2EDuration="1.732844783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:24.730755792 +0000 UTC m=+1012.824922058" watchObservedRunningTime="2026-09-15 07:35:24.732844783 +0000 UTC m=+1012.827011049"
Sep 15 07:35:24.860400 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.860374    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:24.960491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.960424    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gwq6\" (UniqueName: \"kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6\") pod \"2f250fa1-038a-4b53-a2ef-70b410a6135a\" (UID: \"2f250fa1-038a-4b53-a2ef-70b410a6135a\") "
Sep 15 07:35:24.962589 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:24.962552    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6" (OuterVolumeSpecName: "kube-api-access-5gwq6") pod "2f250fa1-038a-4b53-a2ef-70b410a6135a" (UID: "2f250fa1-038a-4b53-a2ef-70b410a6135a"). InnerVolumeSpecName "kube-api-access-5gwq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:35:25.061530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.061497    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gwq6\" (UniqueName: \"kubernetes.io/projected/2f250fa1-038a-4b53-a2ef-70b410a6135a-kube-api-access-5gwq6\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:35:25.206282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.206251    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:25.211497 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.211443    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:25.691986 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.691949    2574 generic.go:356] "Generic (PLEG): container finished" podID="2f250fa1-038a-4b53-a2ef-70b410a6135a" containerID="52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308" exitCode=137
Sep 15 07:35:25.692420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.692002    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:25.692420 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.692052    2574 scope.go:117] "RemoveContainer" containerID="52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308"
Sep 15 07:35:25.692638 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.692620    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:25.693645 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.693622    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:25.703596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.703555    2574 scope.go:117] "RemoveContainer" containerID="52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308"
Sep 15 07:35:25.703845 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:25.703824    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308\": container with ID starting with 52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308 not found: ID does not exist" containerID="52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308"
Sep 15 07:35:25.703925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:25.703853    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308"} err="failed to get container status \"52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308\": rpc error: code = NotFound desc = could not find container \"52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308\": container with ID starting with 52ddfbdec22bebe32955ef090736f629a9a630796d58d82ef250fc00aece8308 not found: ID does not exist"
Sep 15 07:35:25.719442 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:25.719412    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7669-95c529\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" pod="opendatahub/test-curl-7669-95c529"
Sep 15 07:35:26.380965 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:26.380921    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f250fa1-038a-4b53-a2ef-70b410a6135a" path="/var/lib/kubelet/pods/2f250fa1-038a-4b53-a2ef-70b410a6135a/volumes"
Sep 15 07:35:28.687465 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.687432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:35:28.688066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.688043    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:35:28.689862 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.689834    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:35:28.879142 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.870851    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:35:28.906703 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.906635    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:35:28.906878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.906793    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:28.922156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.920353    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e8faa2dc-serving-cert\""
Sep 15 07:35:28.922156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.921516    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:28.922156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.921590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrq9\" (UniqueName: \"kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:28.922156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.921653    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:28.922156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:28.921699    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.023139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.023036    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.023139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.023085    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.023339 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.023143    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrq9\" (UniqueName: \"kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.023339 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:29.023248    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-serving-cert: secret "maas-api-e2e-ait-e8faa2dc-serving-cert" not found
Sep 15 07:35:29.023339 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:29.023315    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls podName:309f4a85-26a1-4ba7-968b-e5cafb14b30c nodeName:}" failed. No retries permitted until 2026-09-15 07:35:29.523299268 +0000 UTC m=+1017.617465508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls") pod "maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" (UID: "309f4a85-26a1-4ba7-968b-e5cafb14b30c") : secret "maas-api-e2e-ait-e8faa2dc-serving-cert" not found
Sep 15 07:35:29.023482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.023408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.023941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.023923    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.025520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.025499    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.036005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.035978    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrq9\" (UniqueName: \"kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.528541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.528510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.530780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.530752    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") pod \"maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.532602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.532583    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:29.874709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:29.873570    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:35:30.737190 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:30.737159    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" event={"ID":"309f4a85-26a1-4ba7-968b-e5cafb14b30c","Type":"ContainerStarted","Data":"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"}
Sep 15 07:35:30.737407 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:30.737218    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" event={"ID":"309f4a85-26a1-4ba7-968b-e5cafb14b30c","Type":"ContainerStarted","Data":"f2cb4f5e5e9e79ad3b6ea657d152fc3b1f16d7558622097121b72c2a63d79e55"}
Sep 15 07:35:30.737407 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:30.737247    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:31.151723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.151602    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" podStartSLOduration=2.64563676 podStartE2EDuration="3.151577336s" totalImagesPullingTime="505.940576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:30.758799454 +0000 UTC m=+1018.852965718" watchObservedRunningTime="2026-09-15 07:35:31.151577336 +0000 UTC m=+1019.245743600"
Sep 15 07:35:31.153544 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.153515    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:35:31.157697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.157670    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.159695 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.159666    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f33ec8-serving-cert\""
Sep 15 07:35:31.179658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.179632    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:35:31.243334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.243309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.243634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.243617    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsxw7\" (UniqueName: \"kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.243799 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.243765    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.243902 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.243893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.345176 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.345147    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsxw7\" (UniqueName: \"kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.345364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.345200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.345364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.345227    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.345364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.345257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.345364 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:31.345360    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-serving-cert: secret "maas-api-e2e-autores-f33ec8-serving-cert" not found
Sep 15 07:35:31.345592 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:31.345407    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls podName:c2c95adf-6157-457b-a676-1b6b2c0bda81 nodeName:}" failed. No retries permitted until 2026-09-15 07:35:31.845391707 +0000 UTC m=+1019.939557946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls") pod "maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" (UID: "c2c95adf-6157-457b-a676-1b6b2c0bda81") : secret "maas-api-e2e-autores-f33ec8-serving-cert" not found
Sep 15 07:35:31.346059 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.346031    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.358812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.356718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.360905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.360880    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsxw7\" (UniqueName: \"kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.852870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.852818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:31.857875 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:31.857850    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") pod \"maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:32.075309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:32.075270    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:32.444727 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:32.444685    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:35:32.446108 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:35:32.446085    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c95adf_6157_457b_a676_1b6b2c0bda81.slice/crio-f9314318789e3aea3c3167d74f380d960a51539244a3a097bfd09109e6700f42 WatchSource:0}: Error finding container f9314318789e3aea3c3167d74f380d960a51539244a3a097bfd09109e6700f42: Status 404 returned error can't find the container with id f9314318789e3aea3c3167d74f380d960a51539244a3a097bfd09109e6700f42
Sep 15 07:35:32.760281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:32.760240    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" event={"ID":"c2c95adf-6157-457b-a676-1b6b2c0bda81","Type":"ContainerStarted","Data":"f9314318789e3aea3c3167d74f380d960a51539244a3a097bfd09109e6700f42"}
Sep 15 07:35:34.775396 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:34.775348    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" event={"ID":"c2c95adf-6157-457b-a676-1b6b2c0bda81","Type":"ContainerStarted","Data":"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"}
Sep 15 07:35:34.775854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:34.775446    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:34.794489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:34.794438    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" podStartSLOduration=1.911366551 podStartE2EDuration="3.794424815s" totalImagesPullingTime="1.883058264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:34.792476032 +0000 UTC m=+1022.886642295" watchObservedRunningTime="2026-09-15 07:35:34.794424815 +0000 UTC m=+1022.888591077"
Sep 15 07:35:36.749516 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.749484    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:35:36.790316 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.790282    2574 generic.go:356] "Generic (PLEG): container finished" podID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerID="35ca8dfb8146b8fdc977efb966b2fd77e4f1a9d79b637ccd86f1e06c82e8852b" exitCode=0
Sep 15 07:35:36.790503 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.790358    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerDied","Data":"35ca8dfb8146b8fdc977efb966b2fd77e4f1a9d79b637ccd86f1e06c82e8852b"}
Sep 15 07:35:36.791033 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.790859    2574 scope.go:117] "RemoveContainer" containerID="35ca8dfb8146b8fdc977efb966b2fd77e4f1a9d79b637ccd86f1e06c82e8852b"
Sep 15 07:35:36.792471 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792443    2574 generic.go:356] "Generic (PLEG): container finished" podID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerID="db1f3ef91cc70e6198d040932ae2679b041f1acff5f25d8fd79e50229ecb9f9d" exitCode=0
Sep 15 07:35:36.792573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792460    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:36.792573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerDied","Data":"db1f3ef91cc70e6198d040932ae2679b041f1acff5f25d8fd79e50229ecb9f9d"}
Sep 15 07:35:36.792573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792548    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:36.792757 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792595    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:36.792987 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.792958    2574 scope.go:117] "RemoveContainer" containerID="db1f3ef91cc70e6198d040932ae2679b041f1acff5f25d8fd79e50229ecb9f9d"
Sep 15 07:35:36.794331 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.794306    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:36.794429 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.794375    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:36.794429 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:36.794424    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:37.800425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:37.800380    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerStarted","Data":"f300d330361acfdd4fb3bd627d472cc564028ee938ac12c78aeee8c29724c5c7"}
Sep 15 07:35:37.802411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:37.802388    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerStarted","Data":"3f33e09acc72bffb6129cf6e42cc39aceb50b350a741452c0c3b12a3fa57a99d"}
Sep 15 07:35:38.594564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.594521    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:38.594779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.594668    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:38.594779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.594736    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:38.662140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.659480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:38.662140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.659837    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:38.662140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.659990    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:38.667142 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.663799    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:35:38.702246 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.702218    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:35:38.702545 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.702530    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.724157 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.719320    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fcd453-serving-cert\""
Sep 15 07:35:38.801661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.801615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.802278 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.802246    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.802514 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.802495    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjgf\" (UniqueName: \"kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.802657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.802641    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.903586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.903493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.903867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.903842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.904044 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:38.904013    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-serving-cert: secret "maas-api-e2e-inf-b-fcd453-serving-cert" not found
Sep 15 07:35:38.904184 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:35:38.904138    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls podName:7358ceb2-d8d3-4c11-bbe2-974b491e180d nodeName:}" failed. No retries permitted until 2026-09-15 07:35:39.404088191 +0000 UTC m=+1027.498254454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls") pod "maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" (UID: "7358ceb2-d8d3-4c11-bbe2-974b491e180d") : secret "maas-api-e2e-inf-b-fcd453-serving-cert" not found
Sep 15 07:35:38.904256 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.904242    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjgf\" (UniqueName: \"kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.904749 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.904719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.905502 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.905474    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.906656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.906636    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:38.913301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:38.913277    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjgf\" (UniqueName: \"kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:39.410909 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.410736    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:39.413450 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.413362    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") pod \"maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:39.595560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.595523    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:39.595819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.595788    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:39.634908 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.634490    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:39.662745 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.662667    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:39.662874 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.662769    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:39.725904 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:39.725536    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:35:40.010820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.010718    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:35:40.599860 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.599819    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:40.599972 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.599882    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:40.661588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.661555    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:40.661700 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.661613    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:40.790829 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.790790    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:35:40.843390 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.843309    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" event={"ID":"7358ceb2-d8d3-4c11-bbe2-974b491e180d","Type":"ContainerStarted","Data":"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"}
Sep 15 07:35:40.843596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.843578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" event={"ID":"7358ceb2-d8d3-4c11-bbe2-974b491e180d","Type":"ContainerStarted","Data":"1e17191464d1416a51b880d422cd4fbc810dc2c3dad8485c65778649d53057b7"}
Sep 15 07:35:40.843689 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.843679    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:40.870139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:40.860734    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" podStartSLOduration=2.320880009 podStartE2EDuration="2.86071386s" totalImagesPullingTime="539.833851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:35:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:35:40.85845664 +0000 UTC m=+1028.952622903" watchObservedRunningTime="2026-09-15 07:35:40.86071386 +0000 UTC m=+1028.954880125"
Sep 15 07:35:41.594191 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:41.594074    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:41.594756 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:41.594720    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:41.659889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:41.659449    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:41.659889 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:41.659514    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:42.597141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:42.596852    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:42.597141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:42.596911    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:42.659272 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:42.659225    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:42.659462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:42.659296    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:43.594384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:43.594344    2574 patch_prober.go:28] interesting pod/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body=
Sep 15 07:35:43.594661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:43.594632    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused"
Sep 15 07:35:43.660069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:43.660032    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 15 07:35:43.660674 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:43.660625    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 15 07:35:44.599500 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.599463    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:44.599997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.599975    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:44.600675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.600649    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:35:44.672712 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.671365    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:44.672712 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.672565    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:44.685892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:44.685832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:35:46.862064 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:46.862032    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:35:50.914905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.914875    2574 generic.go:356] "Generic (PLEG): container finished" podID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerID="4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a" exitCode=0
Sep 15 07:35:50.915452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.914950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerDied","Data":"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"}
Sep 15 07:35:50.915452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.915364    2574 scope.go:117] "RemoveContainer" containerID="4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"
Sep 15 07:35:50.916597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.916562    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:50.916707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.916651    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:50.916793 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:50.916718    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:35:51.922850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:51.922810    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerStarted","Data":"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"}
Sep 15 07:35:52.205868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:52.205778    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:52.206867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:52.206838    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:52.207000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:52.206899    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:53.206477 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:53.206438    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:53.206966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:53.206505    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:54.206581 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:54.206549    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:54.206968 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:54.206607    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:55.206626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:55.206588    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:55.207017 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:55.206646    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:55.693467 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:55.693431    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:56.206135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:56.206082    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:56.206325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:56.206189    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:57.206675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:57.206640    2574 patch_prober.go:28] interesting pod/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body=
Sep 15 07:35:57.207192 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:57.206716    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused"
Sep 15 07:35:58.209919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:58.209889    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:35:58.210691 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:35:58.210671    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:36:00.601094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:00.601062    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:36:07.423675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.423641    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:36:07.424104 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.423846    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" podUID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" containerName="maas-api" containerID="cri-o://1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923" gracePeriod=30
Sep 15 07:36:07.678869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.678805    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:36:07.762890 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.762846    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:36:07.763288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.763233    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy" containerID="cri-o://e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc" gracePeriod=30
Sep 15 07:36:07.811670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.811637    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") pod \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") "
Sep 15 07:36:07.811786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.811688    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nrq9\" (UniqueName: \"kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9\") pod \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") "
Sep 15 07:36:07.811786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.811750    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs\") pod \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") "
Sep 15 07:36:07.811786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.811780    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca\") pod \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\" (UID: \"309f4a85-26a1-4ba7-968b-e5cafb14b30c\") "
Sep 15 07:36:07.812212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.812184    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "309f4a85-26a1-4ba7-968b-e5cafb14b30c" (UID: "309f4a85-26a1-4ba7-968b-e5cafb14b30c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:07.813992 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.813966    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "309f4a85-26a1-4ba7-968b-e5cafb14b30c" (UID: "309f4a85-26a1-4ba7-968b-e5cafb14b30c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:36:07.814235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.814218    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9" (OuterVolumeSpecName: "kube-api-access-6nrq9") pod "309f4a85-26a1-4ba7-968b-e5cafb14b30c" (UID: "309f4a85-26a1-4ba7-968b-e5cafb14b30c"). InnerVolumeSpecName "kube-api-access-6nrq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:07.814302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.814273    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "309f4a85-26a1-4ba7-968b-e5cafb14b30c" (UID: "309f4a85-26a1-4ba7-968b-e5cafb14b30c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:36:07.913430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.913401    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.913430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.913427    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrq9\" (UniqueName: \"kubernetes.io/projected/309f4a85-26a1-4ba7-968b-e5cafb14b30c-kube-api-access-6nrq9\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.913430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.913437    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/309f4a85-26a1-4ba7-968b-e5cafb14b30c-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:07.913430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:07.913446    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/309f4a85-26a1-4ba7-968b-e5cafb14b30c-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:08.018300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.018215    2574 generic.go:356] "Generic (PLEG): container finished" podID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" containerID="1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923" exitCode=0
Sep 15 07:36:08.018300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.018266    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" event={"ID":"309f4a85-26a1-4ba7-968b-e5cafb14b30c","Type":"ContainerDied","Data":"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"}
Sep 15 07:36:08.018486 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.018302    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8" event={"ID":"309f4a85-26a1-4ba7-968b-e5cafb14b30c","Type":"ContainerDied","Data":"f2cb4f5e5e9e79ad3b6ea657d152fc3b1f16d7558622097121b72c2a63d79e55"}
Sep 15 07:36:08.018486 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.018323    2574 scope.go:117] "RemoveContainer" containerID="1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"
Sep 15 07:36:08.018486 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.018266    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"
Sep 15 07:36:08.027933 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.027916    2574 scope.go:117] "RemoveContainer" containerID="1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"
Sep 15 07:36:08.028210 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:08.028191    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923\": container with ID starting with 1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923 not found: ID does not exist" containerID="1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"
Sep 15 07:36:08.028267 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.028218    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923"} err="failed to get container status \"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923\": rpc error: code = NotFound desc = could not find container \"1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923\": container with ID starting with 1da59d7f165dc9abf39d0436cdb7ec9071d98be5bd84d4f91cf2f97e87c39923 not found: ID does not exist"
Sep 15 07:36:08.037979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.037957    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:36:08.041599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.041580    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e8faa2dc-7b646c6c-b6vf8"]
Sep 15 07:36:08.379228 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.379154    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" path="/var/lib/kubelet/pods/309f4a85-26a1-4ba7-968b-e5cafb14b30c/volumes"
Sep 15 07:36:08.642366 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.642288    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:08.642879 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.642864    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" containerName="maas-api"
Sep 15 07:36:08.642923 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.642883    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" containerName="maas-api"
Sep 15 07:36:08.643007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.642996    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="309f4a85-26a1-4ba7-968b-e5cafb14b30c" containerName="maas-api"
Sep 15 07:36:08.646232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.646213    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.648391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.648370    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4e9b0a83-gw-openshift-default-dockercfg-gnm52\""
Sep 15 07:36:08.657046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.657026    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:08.721187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721159    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721207    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721226    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721388 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721300    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721388 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ghjk\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721411    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721435    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.721518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.721507    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822451    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822561    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822599    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ghjk\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822628    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.822666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822648    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822915    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822952    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.822997    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823238 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.823136    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.823444 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.823425    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.825027 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.824999    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.825318 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.825299    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.830336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.830314    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.830672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.830653    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ghjk\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk\") pod \"e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") " pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:08.959787 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:08.959757    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:09.236810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:09.236783    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:09.237675 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:36:09.237643    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49813bcb_9bea_46b8_8402_6dd27e6bac4b.slice/crio-d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4 WatchSource:0}: Error finding container d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4: Status 404 returned error can't find the container with id d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4
Sep 15 07:36:09.240301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:09.240264    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:09.240378 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:09.240356    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:09.240419 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:09.240400    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:10.035221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:10.035185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerStarted","Data":"02f886abd0da2fbb2a8408857cab0fa12fdcbf486f4f71562015b2408d107161"}
Sep 15 07:36:10.035606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:10.035264    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerStarted","Data":"d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4"}
Sep 15 07:36:10.082713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:10.082665    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podStartSLOduration=2.082652003 podStartE2EDuration="2.082652003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:10.079776919 +0000 UTC m=+1058.173943204" watchObservedRunningTime="2026-09-15 07:36:10.082652003 +0000 UTC m=+1058.176818265"
Sep 15 07:36:10.960195 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:10.960161    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:10.965490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:10.965466    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:11.041203 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:11.041172    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:11.041872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:11.041853    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:12.708817 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:12.708786    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:36:13.006747 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.006648    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:13.014882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.013915    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.016656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.016615    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:13.017110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.016918    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4e9b0a83-serving-cert\""
Sep 15 07:36:13.035364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.035337    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:36:13.057502 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.057476    2574 generic.go:356] "Generic (PLEG): container finished" podID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerID="e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc" exitCode=0
Sep 15 07:36:13.057640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.057537    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerDied","Data":"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"}
Sep 15 07:36:13.057640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.057575    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"
Sep 15 07:36:13.057640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.057595    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z" event={"ID":"b9960ee4-ff91-4514-bb4a-8a32a90646ab","Type":"ContainerDied","Data":"e8f9b898ff2683556e7204c71ca7201d16a5cb0836494e19a2e383f4e44ed464"}
Sep 15 07:36:13.057640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.057628    2574 scope.go:117] "RemoveContainer" containerID="e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"
Sep 15 07:36:13.062679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.062651    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.062880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.062710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.062880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.062741    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.062880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.062861    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpk5w\" (UniqueName: \"kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.068745 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.068707    2574 scope.go:117] "RemoveContainer" containerID="4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"
Sep 15 07:36:13.077481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.077464    2574 scope.go:117] "RemoveContainer" containerID="e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"
Sep 15 07:36:13.077784 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:13.077763    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc\": container with ID starting with e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc not found: ID does not exist" containerID="e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"
Sep 15 07:36:13.077878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.077798    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc"} err="failed to get container status \"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc\": rpc error: code = NotFound desc = could not find container \"e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc\": container with ID starting with e62c1daf6680d76cfc4d6d5c7618d78b9ba976cdf59a628fa56b4c74d3672acc not found: ID does not exist"
Sep 15 07:36:13.077878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.077815    2574 scope.go:117] "RemoveContainer" containerID="4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"
Sep 15 07:36:13.078131 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:13.078079    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a\": container with ID starting with 4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a not found: ID does not exist" containerID="4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"
Sep 15 07:36:13.078226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.078137    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a"} err="failed to get container status \"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a\": rpc error: code = NotFound desc = could not find container \"4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a\": container with ID starting with 4a4603d077eb4bc853dbd17c8abf4160b6119b73fc41edd8bfdf39931d84c31a not found: ID does not exist"
Sep 15 07:36:13.164021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.163988    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164208 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164044    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164208 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164087    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164208 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164103    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nsjq\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164208 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164158    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164271    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:13.164417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164293    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164338    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164371    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy\") pod \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\" (UID: \"b9960ee4-ff91-4514-bb4a-8a32a90646ab\") "
Sep 15 07:36:13.164664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164440    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:13.164664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164477    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:13.164664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.164824 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.164824 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.164928 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpk5w\" (UniqueName: \"kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.165397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.165052    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.165397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.165073    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.165397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.165088    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.165397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.164547    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:13.165689 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.165439    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.165903 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:13.165820    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-serving-cert: secret "maas-api-e2e-migrate-4e9b0a83-serving-cert" not found
Sep 15 07:36:13.165979 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:13.165910    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls podName:dd8da7b0-f741-4219-9560-09cf5353e416 nodeName:}" failed. No retries permitted until 2026-09-15 07:36:13.665889155 +0000 UTC m=+1061.760055406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls") pod "maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" (UID: "dd8da7b0-f741-4219-9560-09cf5353e416") : secret "maas-api-e2e-migrate-4e9b0a83-serving-cert" not found
Sep 15 07:36:13.166246 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.166222    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:13.167032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.167007    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:13.167141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.167067    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq" (OuterVolumeSpecName: "kube-api-access-9nsjq") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "kube-api-access-9nsjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:13.167141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.167077    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:36:13.167141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.167094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9960ee4-ff91-4514-bb4a-8a32a90646ab" (UID: "b9960ee4-ff91-4514-bb4a-8a32a90646ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:13.168370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.168348    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.173180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.173159    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpk5w\" (UniqueName: \"kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.265943    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.265968    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.265979    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.265990    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nsjq\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-kube-api-access-9nsjq\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.266000    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9960ee4-ff91-4514-bb4a-8a32a90646ab-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.266024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.266008    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9960ee4-ff91-4514-bb4a-8a32a90646ab-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:13.384512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.384477    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:36:13.392923 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.392893    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e8faa2dc-openshift-default-b54449cbd-vp75z"]
Sep 15 07:36:13.669998 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.669962    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.672922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.672891    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") pod \"maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:13.942533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:13.942455    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:14.216893 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:14.216866    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:14.217890 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:36:14.217862    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8da7b0_f741_4219_9560_09cf5353e416.slice/crio-ce1b30ba193b11185c008e6e483257c58dd7e9d7798974a25c3fbdd0ca1bf98d WatchSource:0}: Error finding container ce1b30ba193b11185c008e6e483257c58dd7e9d7798974a25c3fbdd0ca1bf98d: Status 404 returned error can't find the container with id ce1b30ba193b11185c008e6e483257c58dd7e9d7798974a25c3fbdd0ca1bf98d
Sep 15 07:36:14.379032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:14.379002    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" path="/var/lib/kubelet/pods/b9960ee4-ff91-4514-bb4a-8a32a90646ab/volumes"
Sep 15 07:36:15.071364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:15.071338    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" event={"ID":"dd8da7b0-f741-4219-9560-09cf5353e416","Type":"ContainerStarted","Data":"ce1b30ba193b11185c008e6e483257c58dd7e9d7798974a25c3fbdd0ca1bf98d"}
Sep 15 07:36:16.078554 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:16.078520    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" event={"ID":"dd8da7b0-f741-4219-9560-09cf5353e416","Type":"ContainerStarted","Data":"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"}
Sep 15 07:36:16.079052 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:16.078575    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:16.098272 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:16.098223    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" podStartSLOduration=3.304590225 podStartE2EDuration="4.098207532s" totalImagesPullingTime="793.617307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:16.096385998 +0000 UTC m=+1064.190552260" watchObservedRunningTime="2026-09-15 07:36:16.098207532 +0000 UTC m=+1064.192373794"
Sep 15 07:36:22.089239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:22.089209    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:30.901927 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:30.901895    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:36:33.183462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.183428    2574 generic.go:356] "Generic (PLEG): container finished" podID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerID="02f886abd0da2fbb2a8408857cab0fa12fdcbf486f4f71562015b2408d107161" exitCode=0
Sep 15 07:36:33.183949 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.183506    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerDied","Data":"02f886abd0da2fbb2a8408857cab0fa12fdcbf486f4f71562015b2408d107161"}
Sep 15 07:36:33.184043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.184025    2574 scope.go:117] "RemoveContainer" containerID="02f886abd0da2fbb2a8408857cab0fa12fdcbf486f4f71562015b2408d107161"
Sep 15 07:36:33.185209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.185177    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:33.185323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.185246    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:33.185323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:33.185282    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:36:34.192419 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:34.192387    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerStarted","Data":"5d9aa50df8476bc2d1242e03a71b5137def27de125e00e850b0a17b4ddece136"}
Sep 15 07:36:34.960084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:34.960041    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:34.960308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:34.960271    2574 patch_prober.go:28] interesting pod/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 15 07:36:34.960384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:34.960352    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 15 07:36:35.961161 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:35.961103    2574 patch_prober.go:28] interesting pod/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 15 07:36:35.961716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:35.961684    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 15 07:36:36.961078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:36.961030    2574 patch_prober.go:28] interesting pod/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 15 07:36:36.961295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:36.961101    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 15 07:36:37.961094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:37.961053    2574 patch_prober.go:28] interesting pod/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 15 07:36:37.961424 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:37.961389    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 15 07:36:38.960330 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:38.960293    2574 patch_prober.go:28] interesting pod/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body=
Sep 15 07:36:38.960712 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:38.960683    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused"
Sep 15 07:36:39.964944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:39.964910    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:39.965377 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:39.965254    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:39.965918 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:39.965899    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:42.664295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.664262    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:42.664786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.664489    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" podUID="dd8da7b0-f741-4219-9560-09cf5353e416" containerName="maas-api" containerID="cri-o://30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259" gracePeriod=30
Sep 15 07:36:42.915835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.915781    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:42.941066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.941042    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca\") pod \"dd8da7b0-f741-4219-9560-09cf5353e416\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") "
Sep 15 07:36:42.941196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.941093    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpk5w\" (UniqueName: \"kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w\") pod \"dd8da7b0-f741-4219-9560-09cf5353e416\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") "
Sep 15 07:36:42.941196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.941165    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") pod \"dd8da7b0-f741-4219-9560-09cf5353e416\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") "
Sep 15 07:36:42.941284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.941214    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs\") pod \"dd8da7b0-f741-4219-9560-09cf5353e416\" (UID: \"dd8da7b0-f741-4219-9560-09cf5353e416\") "
Sep 15 07:36:42.941580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.941543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dd8da7b0-f741-4219-9560-09cf5353e416" (UID: "dd8da7b0-f741-4219-9560-09cf5353e416"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:42.943200 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.943159    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w" (OuterVolumeSpecName: "kube-api-access-gpk5w") pod "dd8da7b0-f741-4219-9560-09cf5353e416" (UID: "dd8da7b0-f741-4219-9560-09cf5353e416"). InnerVolumeSpecName "kube-api-access-gpk5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:42.943284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.943247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dd8da7b0-f741-4219-9560-09cf5353e416" (UID: "dd8da7b0-f741-4219-9560-09cf5353e416"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:36:42.943284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:42.943263    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dd8da7b0-f741-4219-9560-09cf5353e416" (UID: "dd8da7b0-f741-4219-9560-09cf5353e416"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:36:43.019261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.019221    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:43.019538 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.019479    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy" containerID="cri-o://5d9aa50df8476bc2d1242e03a71b5137def27de125e00e850b0a17b4ddece136" gracePeriod=30
Sep 15 07:36:43.042720 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.042694    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8da7b0-f741-4219-9560-09cf5353e416-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.042720 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.042719    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpk5w\" (UniqueName: \"kubernetes.io/projected/dd8da7b0-f741-4219-9560-09cf5353e416-kube-api-access-gpk5w\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.042878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.042728    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.042878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.042737    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8da7b0-f741-4219-9560-09cf5353e416-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:43.246029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.245944    2574 generic.go:356] "Generic (PLEG): container finished" podID="dd8da7b0-f741-4219-9560-09cf5353e416" containerID="30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259" exitCode=0
Sep 15 07:36:43.246029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.245994    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"
Sep 15 07:36:43.246244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.246033    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" event={"ID":"dd8da7b0-f741-4219-9560-09cf5353e416","Type":"ContainerDied","Data":"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"}
Sep 15 07:36:43.246244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.246092    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57" event={"ID":"dd8da7b0-f741-4219-9560-09cf5353e416","Type":"ContainerDied","Data":"ce1b30ba193b11185c008e6e483257c58dd7e9d7798974a25c3fbdd0ca1bf98d"}
Sep 15 07:36:43.246244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.246143    2574 scope.go:117] "RemoveContainer" containerID="30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"
Sep 15 07:36:43.258476 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.258453    2574 scope.go:117] "RemoveContainer" containerID="30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"
Sep 15 07:36:43.258795 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:43.258771    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259\": container with ID starting with 30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259 not found: ID does not exist" containerID="30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"
Sep 15 07:36:43.258861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.258806    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259"} err="failed to get container status \"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259\": rpc error: code = NotFound desc = could not find container \"30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259\": container with ID starting with 30c6f6e4430d19188c1637efb8f37e9c6e25ab7723a258d5c521959a4818b259 not found: ID does not exist"
Sep 15 07:36:43.270370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.270344    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:43.275169 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:43.275147    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4e9b0a83-8657d6b4bd-2zt57"]
Sep 15 07:36:44.379921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:44.379886    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8da7b0-f741-4219-9560-09cf5353e416" path="/var/lib/kubelet/pods/dd8da7b0-f741-4219-9560-09cf5353e416/volumes"
Sep 15 07:36:48.277430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.277396    2574 generic.go:356] "Generic (PLEG): container finished" podID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerID="5d9aa50df8476bc2d1242e03a71b5137def27de125e00e850b0a17b4ddece136" exitCode=0
Sep 15 07:36:48.277814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.277475    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerDied","Data":"5d9aa50df8476bc2d1242e03a71b5137def27de125e00e850b0a17b4ddece136"}
Sep 15 07:36:48.277814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.277586    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t" event={"ID":"49813bcb-9bea-46b8-8402-6dd27e6bac4b","Type":"ContainerDied","Data":"d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4"}
Sep 15 07:36:48.277814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.277637    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d27f019a2361cd4e408867a9190245f350855bf02698074989b22c411834e2a4"
Sep 15 07:36:48.277814 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.277667    2574 scope.go:117] "RemoveContainer" containerID="02f886abd0da2fbb2a8408857cab0fa12fdcbf486f4f71562015b2408d107161"
Sep 15 07:36:48.285351 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.285303    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:48.318603 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.318578    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:36:48.319054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319035    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319058    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319075    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319081    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319093    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8da7b0-f741-4219-9560-09cf5353e416" containerName="maas-api"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319098    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8da7b0-f741-4219-9560-09cf5353e416" containerName="maas-api"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319106    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.319148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319127    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.319513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319208    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:36:48.319513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319216    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.319513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319222    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8da7b0-f741-4219-9560-09cf5353e416" containerName="maas-api"
Sep 15 07:36:48.319513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.319229    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" containerName="istio-proxy"
Sep 15 07:36:48.321855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.321838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.324869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.324843    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b63bda62-serving-cert\""
Sep 15 07:36:48.330859 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.330668    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:36:48.388078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388049    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388097    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388190    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ghjk\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388210    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388227    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388261    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388345    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388375    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388400    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token\") pod \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\" (UID: \"49813bcb-9bea-46b8-8402-6dd27e6bac4b\") "
Sep 15 07:36:48.388588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388485    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data" (OuterVolumeSpecName: "istio-data") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:48.388588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:48.388761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388625    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:48.388761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq62c\" (UniqueName: \"kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.388761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388716    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:48.388916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388835    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.388916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.389013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.388942    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:36:48.389056 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.389186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389169    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.389263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389191    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.389263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389206    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.389263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389220    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.389263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.389233    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.390508 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.390482    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk" (OuterVolumeSpecName: "kube-api-access-8ghjk") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "kube-api-access-8ghjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:48.390603 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.390515    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:36:48.390698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.390684    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:36:48.390819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.390799    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token" (OuterVolumeSpecName: "istio-token") pod "49813bcb-9bea-46b8-8402-6dd27e6bac4b" (UID: "49813bcb-9bea-46b8-8402-6dd27e6bac4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:36:48.490280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.490437 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490287    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.490437 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.490437 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq62c\" (UniqueName: \"kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490483    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490500    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ghjk\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-kube-api-access-8ghjk\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490514    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490529    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49813bcb-9bea-46b8-8402-6dd27e6bac4b-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:48.490539    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-serving-cert: secret "maas-api-e2e-ait-b63bda62-serving-cert" not found
Sep 15 07:36:48.490619 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:36:48.490596    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls podName:fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18 nodeName:}" failed. No retries permitted until 2026-09-15 07:36:48.99057892 +0000 UTC m=+1097.084745179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls") pod "maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" (UID: "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18") : secret "maas-api-e2e-ait-b63bda62-serving-cert" not found
Sep 15 07:36:48.490994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.490973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.492601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.492584    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.498033 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.498017    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq62c\" (UniqueName: \"kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.807021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.806988    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:36:48.994716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.994682    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:48.996988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:48.996960    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") pod \"maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:49.235728 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:49.235695    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:49.287733 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:49.287684    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"
Sep 15 07:36:49.336579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:49.336521    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:49.341149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:49.340990    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4e9b0a83-gw-openshift-default-6dc8ff6db8-4mm8t"]
Sep 15 07:36:49.513364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:49.513339    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:36:49.515025 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:36:49.514992    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd817c6b_fbe5_4e4c_b9a0_3363d2b67f18.slice/crio-b149219a2b5c09ba8223526044d64f803580fcaca0869ddd686a6b6c075a571a WatchSource:0}: Error finding container b149219a2b5c09ba8223526044d64f803580fcaca0869ddd686a6b6c075a571a: Status 404 returned error can't find the container with id b149219a2b5c09ba8223526044d64f803580fcaca0869ddd686a6b6c075a571a
Sep 15 07:36:50.294315 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:50.294285    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" event={"ID":"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18","Type":"ContainerStarted","Data":"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"}
Sep 15 07:36:50.294680 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:50.294332    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" event={"ID":"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18","Type":"ContainerStarted","Data":"b149219a2b5c09ba8223526044d64f803580fcaca0869ddd686a6b6c075a571a"}
Sep 15 07:36:50.294680 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:50.294382    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:50.379150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:50.379043    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49813bcb-9bea-46b8-8402-6dd27e6bac4b" path="/var/lib/kubelet/pods/49813bcb-9bea-46b8-8402-6dd27e6bac4b/volumes"
Sep 15 07:36:56.305434 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:56.305403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:36:56.322399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:36:56.322342    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" podStartSLOduration=7.780194484 podStartE2EDuration="8.322324414s" totalImagesPullingTime="542.12993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:36:50.324183385 +0000 UTC m=+1098.418349646" watchObservedRunningTime="2026-09-15 07:36:56.322324414 +0000 UTC m=+1104.416490677"
Sep 15 07:37:07.906213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:07.906180    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:37:18.127087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.127053    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:37:18.127637 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.127286    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" podUID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" containerName="maas-api" containerID="cri-o://376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6" gracePeriod=30
Sep 15 07:37:18.378852 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.378783    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:37:18.463941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.463906    2574 generic.go:356] "Generic (PLEG): container finished" podID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" containerID="376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6" exitCode=0
Sep 15 07:37:18.464141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.463970    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"
Sep 15 07:37:18.464141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.463989    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" event={"ID":"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18","Type":"ContainerDied","Data":"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"}
Sep 15 07:37:18.464141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.464048    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v" event={"ID":"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18","Type":"ContainerDied","Data":"b149219a2b5c09ba8223526044d64f803580fcaca0869ddd686a6b6c075a571a"}
Sep 15 07:37:18.464141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.464073    2574 scope.go:117] "RemoveContainer" containerID="376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"
Sep 15 07:37:18.473261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.473246    2574 scope.go:117] "RemoveContainer" containerID="376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"
Sep 15 07:37:18.473508 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:37:18.473490    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6\": container with ID starting with 376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6 not found: ID does not exist" containerID="376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"
Sep 15 07:37:18.473577 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.473516    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6"} err="failed to get container status \"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6\": rpc error: code = NotFound desc = could not find container \"376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6\": container with ID starting with 376dcc1992cbf49fdbb4ecf5609655a5d41a7b2d467b81d6e67c017ecaabb2c6 not found: ID does not exist"
Sep 15 07:37:18.549798 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.549757    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") pod \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") "
Sep 15 07:37:18.550485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.550444    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs\") pod \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") "
Sep 15 07:37:18.550619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.550513    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq62c\" (UniqueName: \"kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c\") pod \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") "
Sep 15 07:37:18.550619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.550550    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca\") pod \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\" (UID: \"fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18\") "
Sep 15 07:37:18.551270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.551237    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" (UID: "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:18.553105 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.553064    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" (UID: "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:37:18.553839 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.553517    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" (UID: "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:37:18.553839 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.553797    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c" (OuterVolumeSpecName: "kube-api-access-cq62c") pod "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" (UID: "fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18"). InnerVolumeSpecName "kube-api-access-cq62c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:37:18.652232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.652154    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.652232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.652189    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq62c\" (UniqueName: \"kubernetes.io/projected/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-kube-api-access-cq62c\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.652232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.652204    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.652232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.652218    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:18.789245 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.789216    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:37:18.791501 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:18.791480    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b63bda62-b8b8fcdb7-7rq8v"]
Sep 15 07:37:20.189067 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189031    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:37:20.189602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189585    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:37:20.189650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189604    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:37:20.189650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189621    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" containerName="maas-api"
Sep 15 07:37:20.189650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189626    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" containerName="maas-api"
Sep 15 07:37:20.189744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189692    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9960ee4-ff91-4514-bb4a-8a32a90646ab" containerName="istio-proxy"
Sep 15 07:37:20.189744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.189704    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" containerName="maas-api"
Sep 15 07:37:20.192546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.192528    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.194617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.194598    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-67d9aa69-openshift-default-dockercfg-zxzd2\""
Sep 15 07:37:20.205759 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.205738    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:37:20.370455 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370419    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370455 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370456    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sbrr\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370484    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370548    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370584    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370633    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370930 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370768    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.370930 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.370826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.378726 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.378699    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18" path="/var/lib/kubelet/pods/fd817c6b-fbe5-4e4c-b9a0-3363d2b67f18/volumes"
Sep 15 07:37:20.471910 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.471844    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.471910 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.471874    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sbrr\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472021    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472056    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472075    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472105    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472402 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472332    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472402 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472354    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.472402 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472361    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.473562 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.472579    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.473922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.473900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.474057 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.473934    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.474197 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.474176    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.474722 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.474698    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.474937 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.474920    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.480017 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.479994    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sbrr\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.480231 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.480215    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token\") pod \"e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") " pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.502479 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.502460    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:20.783259 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.783237    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:37:20.784569 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:37:20.784541    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4150de06_972d_439a_a198_408438fb7c38.slice/crio-4e20beb11aec263dd53fd5e2ccea70d8295f50db24199ea0315224cd9f97c2a1 WatchSource:0}: Error finding container 4e20beb11aec263dd53fd5e2ccea70d8295f50db24199ea0315224cd9f97c2a1: Status 404 returned error can't find the container with id 4e20beb11aec263dd53fd5e2ccea70d8295f50db24199ea0315224cd9f97c2a1
Sep 15 07:37:20.787203 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.787159    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:20.787319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.787246    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:20.787319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:20.787287    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:21.485256 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:21.485219    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerStarted","Data":"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"}
Sep 15 07:37:21.485625 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:21.485279    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerStarted","Data":"4e20beb11aec263dd53fd5e2ccea70d8295f50db24199ea0315224cd9f97c2a1"}
Sep 15 07:37:21.503175 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:21.503148    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:21.509040 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:21.509017    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:21.510079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:21.510038    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podStartSLOduration=1.510025371 podStartE2EDuration="1.510025371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:37:21.506847998 +0000 UTC m=+1129.601014260" watchObservedRunningTime="2026-09-15 07:37:21.510025371 +0000 UTC m=+1129.604191633"
Sep 15 07:37:22.491344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:22.491311    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:22.492468 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:22.492433    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:26.708515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:26.708483    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:37:28.535662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.535628    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:37:28.538515 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.538493    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.541379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.541350    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-67d9aa69-serving-cert\""
Sep 15 07:37:28.555226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.553561    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:37:28.643804 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.643773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnvt6\" (UniqueName: \"kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.643961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.643874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.643961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.643917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.643961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.643948    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.745212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.745178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.745387 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.745221    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.745387 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.745296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.745387 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.745341    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnvt6\" (UniqueName: \"kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.745559 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:37:28.745393    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-serving-cert: secret "maas-api-e2e-derive-67d9aa69-serving-cert" not found
Sep 15 07:37:28.745559 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:37:28.745468    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls podName:1160cae8-bb8f-487e-aa4d-69bfdb274b0b nodeName:}" failed. No retries permitted until 2026-09-15 07:37:29.245446233 +0000 UTC m=+1137.339612489 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls") pod "maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" (UID: "1160cae8-bb8f-487e-aa4d-69bfdb274b0b") : secret "maas-api-e2e-derive-67d9aa69-serving-cert" not found
Sep 15 07:37:28.745918 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.745900    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.747774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.747742    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:28.754337 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:28.754310    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnvt6\" (UniqueName: \"kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:29.250087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:29.250049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:29.257412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:29.253528    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") pod \"maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:29.461531 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:29.461496    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:29.733701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:29.733665    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:37:29.735132 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:37:29.735087    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1160cae8_bb8f_487e_aa4d_69bfdb274b0b.slice/crio-5b6d192666db788bcab002e0cd7aa48c0a807a9c9a17eca834c15d213f9a81cf WatchSource:0}: Error finding container 5b6d192666db788bcab002e0cd7aa48c0a807a9c9a17eca834c15d213f9a81cf: Status 404 returned error can't find the container with id 5b6d192666db788bcab002e0cd7aa48c0a807a9c9a17eca834c15d213f9a81cf
Sep 15 07:37:30.539489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:30.539402    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" event={"ID":"1160cae8-bb8f-487e-aa4d-69bfdb274b0b","Type":"ContainerStarted","Data":"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"}
Sep 15 07:37:30.539489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:30.539460    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:30.539489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:30.539482    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" event={"ID":"1160cae8-bb8f-487e-aa4d-69bfdb274b0b","Type":"ContainerStarted","Data":"5b6d192666db788bcab002e0cd7aa48c0a807a9c9a17eca834c15d213f9a81cf"}
Sep 15 07:37:30.557187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:30.557144    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" podStartSLOduration=2.109413334 podStartE2EDuration="2.557128365s" totalImagesPullingTime="447.715031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:37:30.556046481 +0000 UTC m=+1138.650212766" watchObservedRunningTime="2026-09-15 07:37:30.557128365 +0000 UTC m=+1138.651294618"
Sep 15 07:37:36.572442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:36.572400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:37:44.202504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.202467    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:37:44.367534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.366837    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:37:44.367534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.367142    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" podUID="c2c95adf-6157-457b-a676-1b6b2c0bda81" containerName="maas-api" containerID="cri-o://fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e" gracePeriod=30
Sep 15 07:37:44.623045 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.623021    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:37:44.627083 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.627058    2574 generic.go:356] "Generic (PLEG): container finished" podID="c2c95adf-6157-457b-a676-1b6b2c0bda81" containerID="fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e" exitCode=0
Sep 15 07:37:44.627206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.627092    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"
Sep 15 07:37:44.627206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.627094    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" event={"ID":"c2c95adf-6157-457b-a676-1b6b2c0bda81","Type":"ContainerDied","Data":"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"}
Sep 15 07:37:44.627317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.627296    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm" event={"ID":"c2c95adf-6157-457b-a676-1b6b2c0bda81","Type":"ContainerDied","Data":"f9314318789e3aea3c3167d74f380d960a51539244a3a097bfd09109e6700f42"}
Sep 15 07:37:44.627374 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.627330    2574 scope.go:117] "RemoveContainer" containerID="fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"
Sep 15 07:37:44.628935 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.628910    2574 generic.go:356] "Generic (PLEG): container finished" podID="4150de06-972d-439a-a198-408438fb7c38" containerID="99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba" exitCode=0
Sep 15 07:37:44.629029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.628943    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerDied","Data":"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"}
Sep 15 07:37:44.629314 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.629300    2574 scope.go:117] "RemoveContainer" containerID="99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"
Sep 15 07:37:44.630548 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.630519    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:44.630637 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.630583    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:44.630637 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.630614    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:37:44.636149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.636101    2574 scope.go:117] "RemoveContainer" containerID="fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"
Sep 15 07:37:44.636399 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:37:44.636379    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e\": container with ID starting with fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e not found: ID does not exist" containerID="fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"
Sep 15 07:37:44.636529 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.636406    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e"} err="failed to get container status \"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e\": rpc error: code = NotFound desc = could not find container \"fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e\": container with ID starting with fc3f111b865765909c95dd5c3f62c2b493f19c3be11b44254409a7e082564c5e not found: ID does not exist"
Sep 15 07:37:44.693415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.693390    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") pod \"c2c95adf-6157-457b-a676-1b6b2c0bda81\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") "
Sep 15 07:37:44.693517 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.693478    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsxw7\" (UniqueName: \"kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7\") pod \"c2c95adf-6157-457b-a676-1b6b2c0bda81\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") "
Sep 15 07:37:44.693597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.693516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs\") pod \"c2c95adf-6157-457b-a676-1b6b2c0bda81\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") "
Sep 15 07:37:44.693597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.693560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca\") pod \"c2c95adf-6157-457b-a676-1b6b2c0bda81\" (UID: \"c2c95adf-6157-457b-a676-1b6b2c0bda81\") "
Sep 15 07:37:44.694317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.694155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c2c95adf-6157-457b-a676-1b6b2c0bda81" (UID: "c2c95adf-6157-457b-a676-1b6b2c0bda81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:37:44.697003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.696969    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c2c95adf-6157-457b-a676-1b6b2c0bda81" (UID: "c2c95adf-6157-457b-a676-1b6b2c0bda81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:37:44.697103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.697004    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7" (OuterVolumeSpecName: "kube-api-access-bsxw7") pod "c2c95adf-6157-457b-a676-1b6b2c0bda81" (UID: "c2c95adf-6157-457b-a676-1b6b2c0bda81"). InnerVolumeSpecName "kube-api-access-bsxw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:37:44.697344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.697323    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c2c95adf-6157-457b-a676-1b6b2c0bda81" (UID: "c2c95adf-6157-457b-a676-1b6b2c0bda81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:37:44.795534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.795388    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:44.795534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.795423    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsxw7\" (UniqueName: \"kubernetes.io/projected/c2c95adf-6157-457b-a676-1b6b2c0bda81-kube-api-access-bsxw7\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:44.795534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.795441    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c2c95adf-6157-457b-a676-1b6b2c0bda81-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:44.795534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.795455    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2c95adf-6157-457b-a676-1b6b2c0bda81-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:37:44.953914 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.953880    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:37:44.956601 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:44.956575    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f33ec8-6dfbfdc48-75rkm"]
Sep 15 07:37:45.638571 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:45.638488    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerStarted","Data":"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"}
Sep 15 07:37:46.380789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:46.380755    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2c95adf-6157-457b-a676-1b6b2c0bda81" path="/var/lib/kubelet/pods/c2c95adf-6157-457b-a676-1b6b2c0bda81/volumes"
Sep 15 07:37:46.502980 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:46.502939    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:46.503216 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:46.503192    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 07:37:46.503304 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:46.503253    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 07:37:47.503438 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:47.503402    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 07:37:47.503812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:47.503471    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 07:37:48.503056 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:48.503016    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 07:37:48.503283 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:48.503079    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 07:37:49.503706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:49.503672    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 07:37:49.504185 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:49.503741    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 07:37:50.503861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:50.503824    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 15 07:37:50.504352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:50.503885    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 15 07:37:51.512054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:51.512024    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:51.512488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:51.512415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:37:51.513474 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:37:51.513448    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:38:05.309587 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.309481    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:38:05.310021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.309768    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" podUID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" containerName="maas-api" containerID="cri-o://00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f" gracePeriod=30
Sep 15 07:38:05.563620 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.563561    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:38:05.692643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.692614    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") pod \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") "
Sep 15 07:38:05.692872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.692828    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs\") pod \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") "
Sep 15 07:38:05.692955 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.692905    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca\") pod \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") "
Sep 15 07:38:05.693017 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.692971    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnvt6\" (UniqueName: \"kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6\") pod \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\" (UID: \"1160cae8-bb8f-487e-aa4d-69bfdb274b0b\") "
Sep 15 07:38:05.693477 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.693449    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1160cae8-bb8f-487e-aa4d-69bfdb274b0b" (UID: "1160cae8-bb8f-487e-aa4d-69bfdb274b0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:38:05.698073 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.698035    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6" (OuterVolumeSpecName: "kube-api-access-jnvt6") pod "1160cae8-bb8f-487e-aa4d-69bfdb274b0b" (UID: "1160cae8-bb8f-487e-aa4d-69bfdb274b0b"). InnerVolumeSpecName "kube-api-access-jnvt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:38:05.698692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.698651    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1160cae8-bb8f-487e-aa4d-69bfdb274b0b" (UID: "1160cae8-bb8f-487e-aa4d-69bfdb274b0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:38:05.699213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.699187    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1160cae8-bb8f-487e-aa4d-69bfdb274b0b" (UID: "1160cae8-bb8f-487e-aa4d-69bfdb274b0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:38:05.737741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.737717    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:38:05.737922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.737903    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" containerID="cri-o://2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3" gracePeriod=30
Sep 15 07:38:05.770401 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.770373    2574 generic.go:356] "Generic (PLEG): container finished" podID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" containerID="00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f" exitCode=0
Sep 15 07:38:05.770518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.770420    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"
Sep 15 07:38:05.770518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.770447    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" event={"ID":"1160cae8-bb8f-487e-aa4d-69bfdb274b0b","Type":"ContainerDied","Data":"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"}
Sep 15 07:38:05.770518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.770497    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt" event={"ID":"1160cae8-bb8f-487e-aa4d-69bfdb274b0b","Type":"ContainerDied","Data":"5b6d192666db788bcab002e0cd7aa48c0a807a9c9a17eca834c15d213f9a81cf"}
Sep 15 07:38:05.770649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.770519    2574 scope.go:117] "RemoveContainer" containerID="00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"
Sep 15 07:38:05.780476 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.780451    2574 scope.go:117] "RemoveContainer" containerID="00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"
Sep 15 07:38:05.780815 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:38:05.780783    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f\": container with ID starting with 00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f not found: ID does not exist" containerID="00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"
Sep 15 07:38:05.780877 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.780821    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f"} err="failed to get container status \"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f\": rpc error: code = NotFound desc = could not find container \"00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f\": container with ID starting with 00128cd07e01d4b92f36c073eefd63f8dd8df93c830150316cbdf0fc20d6ac2f not found: ID does not exist"
Sep 15 07:38:05.794543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.794513    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.794543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.794533    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.794543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.794542    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.794750 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.794551    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnvt6\" (UniqueName: \"kubernetes.io/projected/1160cae8-bb8f-487e-aa4d-69bfdb274b0b-kube-api-access-jnvt6\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:05.794808 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.794787    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:38:05.801154 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:05.801133    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-67d9aa69-777c76d94f-4jwnt"]
Sep 15 07:38:06.379137 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:06.379072    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" path="/var/lib/kubelet/pods/1160cae8-bb8f-487e-aa4d-69bfdb274b0b/volumes"
Sep 15 07:38:06.514775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:06.514734    2574 patch_prober.go:28] interesting pod/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 07:38:06.514927 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:06.514815    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:38:08.003506 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:08.003471    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:38:10.979346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:10.979305    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:38:11.142221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142144    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142221 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142249    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142289    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142322    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142360    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142399    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142398    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:11.142732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142432    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sbrr\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142492    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert\") pod \"4150de06-972d-439a-a198-408438fb7c38\" (UID: \"4150de06-972d-439a-a198-408438fb7c38\") "
Sep 15 07:38:11.142732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142604    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data" (OuterVolumeSpecName: "istio-data") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:11.142732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142627    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:11.142732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142652    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:11.142979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142886    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.142979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142909    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.142979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142922    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.142979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142935    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.142979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.142952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:38:11.144456 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.144436    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token" (OuterVolumeSpecName: "istio-token") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:38:11.144531 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.144513    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:38:11.144835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.144806    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:38:11.144892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.144837    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr" (OuterVolumeSpecName: "kube-api-access-8sbrr") pod "4150de06-972d-439a-a198-408438fb7c38" (UID: "4150de06-972d-439a-a198-408438fb7c38"). InnerVolumeSpecName "kube-api-access-8sbrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:38:11.243621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.243586    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4150de06-972d-439a-a198-408438fb7c38-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.243621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.243619    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4150de06-972d-439a-a198-408438fb7c38-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.243621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.243628    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.243788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.243637    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4150de06-972d-439a-a198-408438fb7c38-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.243788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.243645    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sbrr\" (UniqueName: \"kubernetes.io/projected/4150de06-972d-439a-a198-408438fb7c38-kube-api-access-8sbrr\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:38:11.808701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.808666    2574 generic.go:356] "Generic (PLEG): container finished" podID="4150de06-972d-439a-a198-408438fb7c38" containerID="2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3" exitCode=0
Sep 15 07:38:11.808881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.808715    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"
Sep 15 07:38:11.808881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.808758    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerDied","Data":"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"}
Sep 15 07:38:11.808881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.808815    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx" event={"ID":"4150de06-972d-439a-a198-408438fb7c38","Type":"ContainerDied","Data":"4e20beb11aec263dd53fd5e2ccea70d8295f50db24199ea0315224cd9f97c2a1"}
Sep 15 07:38:11.808881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.808841    2574 scope.go:117] "RemoveContainer" containerID="2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"
Sep 15 07:38:11.818644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.818628    2574 scope.go:117] "RemoveContainer" containerID="99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"
Sep 15 07:38:11.826789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.826774    2574 scope.go:117] "RemoveContainer" containerID="2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"
Sep 15 07:38:11.827041 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:38:11.827023    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3\": container with ID starting with 2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3 not found: ID does not exist" containerID="2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"
Sep 15 07:38:11.827085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.827049    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3"} err="failed to get container status \"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3\": rpc error: code = NotFound desc = could not find container \"2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3\": container with ID starting with 2b5702816ca9a11ff617c2c7d77c5414ae44cb2b937eb5e183dbf5a5720e39f3 not found: ID does not exist"
Sep 15 07:38:11.827085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.827065    2574 scope.go:117] "RemoveContainer" containerID="99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"
Sep 15 07:38:11.827339 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:38:11.827322    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba\": container with ID starting with 99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba not found: ID does not exist" containerID="99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"
Sep 15 07:38:11.827402 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.827350    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba"} err="failed to get container status \"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba\": rpc error: code = NotFound desc = could not find container \"99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba\": container with ID starting with 99cc797d28acc1314514b6408ab077456a64e738e88edd28809506bf900689ba not found: ID does not exist"
Sep 15 07:38:11.831129 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.831095    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:38:11.836306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:11.836286    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-67d9aa69-openshift-default-7dc94f957b-kpstx"]
Sep 15 07:38:12.378935 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:12.378903    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4150de06-972d-439a-a198-408438fb7c38" path="/var/lib/kubelet/pods/4150de06-972d-439a-a198-408438fb7c38/volumes"
Sep 15 07:38:21.617881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:21.617849    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:38:27.709399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709366    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709743    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709754    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709763    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2c95adf-6157-457b-a676-1b6b2c0bda81" containerName="maas-api"
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709768    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2c95adf-6157-457b-a676-1b6b2c0bda81" containerName="maas-api"
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709775    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" containerName="maas-api"
Sep 15 07:38:27.709777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709781    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" containerName="maas-api"
Sep 15 07:38:27.709994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709787    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:27.709994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709793    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:27.709994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709854    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:27.709994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709862    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2c95adf-6157-457b-a676-1b6b2c0bda81" containerName="maas-api"
Sep 15 07:38:27.709994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.709868    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="1160cae8-bb8f-487e-aa4d-69bfdb274b0b" containerName="maas-api"
Sep 15 07:38:27.714338 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.714316    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.717819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.717795    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c258f1-openshift-default-dockercfg-qb4sk\""
Sep 15 07:38:27.728714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.728682    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:38:27.781195 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781236    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781318    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781368    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781417    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxqn5\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781453    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781659 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781476    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.781659 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.781586    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882350 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882358    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882380    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxqn5\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882513 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882616    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882748    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.882883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.883010 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.882904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.883092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.883070    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.883401 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.883378    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.884725 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.884704    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.884944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.884924    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.894548 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.894520    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:27.898693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:27.898667    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxqn5\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5\") pod \"e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") " pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:28.024900 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.024825    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:28.318067 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.318039    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:38:28.319428 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:38:28.319396    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e038515_e9e9_42d7_811d_9238cb31b899.slice/crio-0fecdf0019e4cc1c701778f8435ae535aeda15c641839f8e75f7c89139b78b51 WatchSource:0}: Error finding container 0fecdf0019e4cc1c701778f8435ae535aeda15c641839f8e75f7c89139b78b51: Status 404 returned error can't find the container with id 0fecdf0019e4cc1c701778f8435ae535aeda15c641839f8e75f7c89139b78b51
Sep 15 07:38:28.321542 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.321511    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:28.321596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.321577    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:28.321632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.321606    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:28.905967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.905927    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerStarted","Data":"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"}
Sep 15 07:38:28.906365 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.905994    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerStarted","Data":"0fecdf0019e4cc1c701778f8435ae535aeda15c641839f8e75f7c89139b78b51"}
Sep 15 07:38:28.939783 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:28.939736    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podStartSLOduration=1.939720958 podStartE2EDuration="1.939720958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:38:28.938183338 +0000 UTC m=+1197.032349597" watchObservedRunningTime="2026-09-15 07:38:28.939720958 +0000 UTC m=+1197.033887272"
Sep 15 07:38:29.025796 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:29.025770    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:29.030850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:29.030828    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:29.911800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:29.911773    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:29.912534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:29.912511    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:34.918220 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:34.918182    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:38:34.918946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:34.918919    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4150de06-972d-439a-a198-408438fb7c38" containerName="istio-proxy"
Sep 15 07:38:34.923231 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:34.923207    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:34.927317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:34.927298    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-c258f1-serving-cert\""
Sep 15 07:38:34.934062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:34.934036    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:38:35.046076 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.046045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.046284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.046101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.046284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.046137    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.046284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.046256    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgsd9\" (UniqueName: \"kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.146663 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.146633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgsd9\" (UniqueName: \"kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.146826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.146717    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.146826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.146775    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.146826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.146798    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.147490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.147464    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.149588 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.149539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.149930 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.149909    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.155381 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.155357    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgsd9\" (UniqueName: \"kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9\") pod \"maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.235931 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.235859    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:35.511592 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.511561    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:38:35.512754 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:38:35.512719    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0635fda3_ad6c_4ef9_994e_ffb02b4c0f81.slice/crio-fe647b34c0f42e38395ff2357041b9ae0f4e241455b47ad8e39b523f607ce78f WatchSource:0}: Error finding container fe647b34c0f42e38395ff2357041b9ae0f4e241455b47ad8e39b523f607ce78f: Status 404 returned error can't find the container with id fe647b34c0f42e38395ff2357041b9ae0f4e241455b47ad8e39b523f607ce78f
Sep 15 07:38:35.949616 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:35.949579    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" event={"ID":"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81","Type":"ContainerStarted","Data":"fe647b34c0f42e38395ff2357041b9ae0f4e241455b47ad8e39b523f607ce78f"}
Sep 15 07:38:36.958574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:36.958536    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" event={"ID":"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81","Type":"ContainerStarted","Data":"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"}
Sep 15 07:38:36.959122 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:36.959094    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:36.977639 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:36.977591    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" podStartSLOduration=2.326722368 podStartE2EDuration="2.977576451s" totalImagesPullingTime="650.854083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:38:36.975504979 +0000 UTC m=+1205.069671241" watchObservedRunningTime="2026-09-15 07:38:36.977576451 +0000 UTC m=+1205.071742713"
Sep 15 07:38:39.925807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:39.925769    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:38:42.974754 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:42.974717    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:38:51.047461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.047405    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e038515-e9e9-42d7-811d-9238cb31b899" containerID="8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d" exitCode=0
Sep 15 07:38:51.047963 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.047490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerDied","Data":"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"}
Sep 15 07:38:51.047963 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.047871    2574 scope.go:117] "RemoveContainer" containerID="8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"
Sep 15 07:38:51.049033 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.049005    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:51.049109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.049075    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:51.049194 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:51.049134    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:38:52.057999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:52.057957    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerStarted","Data":"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"}
Sep 15 07:38:53.025056 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:53.025018    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:53.025374 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:53.025251    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 15 07:38:53.025374 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:53.025307    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 15 07:38:54.026068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:54.026037    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 15 07:38:54.026449 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:54.026089    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 15 07:38:55.026164 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:55.026102    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 15 07:38:55.026657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:55.026184    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 15 07:38:56.025649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.025617    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 15 07:38:56.025831 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.025681    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 15 07:38:56.668216 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.668183    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:38:56.680764 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.680711    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.681088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.680859    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:38:56.683524 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.682982    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"default-dockercfg-2jjvm\""
Sep 15 07:38:56.684250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.683987    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"kube-root-ca.crt\""
Sep 15 07:38:56.684485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.684462    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"premium-w0-2735c5-kserve-self-signed-certs\""
Sep 15 07:38:56.684841 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.684663    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"openshift-service-ca.crt\""
Sep 15 07:38:56.842746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.842708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.843029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.843012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.843236 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.843215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.843474 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.843457    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.843658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.843642    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgjx\" (UniqueName: \"kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.843829 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.843814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.944973 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.944897    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945190 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945292 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945278    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945336    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945434    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945514 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945493    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgjx\" (UniqueName: \"kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.945781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.945754    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.948501 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.948434    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.949357 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.949329    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.954692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.954670    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgjx\" (UniqueName: \"kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx\") pod \"premium-w0-2735c5-kserve-7fb7779f89-tlwfp\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") " pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:56.991817 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:56.991797    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:38:57.025428 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:57.025396    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Sep 15 07:38:57.025560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:57.025460    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Sep 15 07:38:57.275812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:57.275716    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:38:57.282048 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:38:57.281967    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7169ef_806d_4ee7_b51c_6145af872f5f.slice/crio-8defe21b35240fb3b3cbe5e47220e386225b1a854ca5b77a2aa825fd308c0767 WatchSource:0}: Error finding container 8defe21b35240fb3b3cbe5e47220e386225b1a854ca5b77a2aa825fd308c0767: Status 404 returned error can't find the container with id 8defe21b35240fb3b3cbe5e47220e386225b1a854ca5b77a2aa825fd308c0767
Sep 15 07:38:58.030424 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:58.030390    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:58.030837 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:58.030821    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:58.031574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:58.031551    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:38:58.107908 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:58.107868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" event={"ID":"2a7169ef-806d-4ee7-b51c-6145af872f5f","Type":"ContainerStarted","Data":"8defe21b35240fb3b3cbe5e47220e386225b1a854ca5b77a2aa825fd308c0767"}
Sep 15 07:38:59.115182 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:59.115139    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" event={"ID":"2a7169ef-806d-4ee7-b51c-6145af872f5f","Type":"ContainerStarted","Data":"fc8c98e66ffc204697fa322719ddc36e4eb7f75f6b376b214bc1e913c5ac70b3"}
Sep 15 07:38:59.147537 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:38:59.147480    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" podStartSLOduration=2.012009637 podStartE2EDuration="3.147458577s" totalImagesPullingTime="1.13544894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:38:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:38:59.143540528 +0000 UTC m=+1227.237706807" watchObservedRunningTime="2026-09-15 07:38:59.147458577 +0000 UTC m=+1227.241624841"
Sep 15 07:39:01.611131 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:01.611089    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:39:06.992975 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:06.992941    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:39:06.993405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:06.992987    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:39:07.005425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:07.005403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:39:07.173802 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:07.173777    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:39:13.734389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.734355    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:39:13.736891 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.736873    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.739895 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.739874    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"default-dockercfg-p295w\""
Sep 15 07:39:13.740202 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.740184    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"openshift-service-ca.crt\""
Sep 15 07:39:13.740580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.740560    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"kube-root-ca.crt\""
Sep 15 07:39:13.740741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.740718    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f56165\"/\"premium-w4-f56165-kserve-self-signed-certs\""
Sep 15 07:39:13.753293 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.753273    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:39:13.800676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800645    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54zh8\" (UniqueName: \"kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.800801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800692    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.800801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.800801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.800801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.800945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.800826    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901726    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901771    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901818    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54zh8\" (UniqueName: \"kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901842    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901859    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.901929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.901876    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.902259 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.902186    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.902259 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.902199    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.902346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.902260    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.904092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.904067    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.904405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.904386    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:13.910547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:13.910523    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54zh8\" (UniqueName: \"kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8\") pod \"premium-w4-f56165-kserve-58df7879-2kcj4\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") " pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:14.046274 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:14.046196    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:14.321186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:14.321157    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:39:14.322368 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:39:14.322341    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb34e7a04_8a74_4f74_bd7b_9d0a6bb17632.slice/crio-2279130790b1aaf450ec243fa057b3980854e860bd879ad11904af03f8bd9dce WatchSource:0}: Error finding container 2279130790b1aaf450ec243fa057b3980854e860bd879ad11904af03f8bd9dce: Status 404 returned error can't find the container with id 2279130790b1aaf450ec243fa057b3980854e860bd879ad11904af03f8bd9dce
Sep 15 07:39:15.215684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:15.215647    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" event={"ID":"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632","Type":"ContainerStarted","Data":"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"}
Sep 15 07:39:15.215684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:15.215691    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" event={"ID":"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632","Type":"ContainerStarted","Data":"2279130790b1aaf450ec243fa057b3980854e860bd879ad11904af03f8bd9dce"}
Sep 15 07:39:15.239684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:15.239630    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" podStartSLOduration=2.239610234 podStartE2EDuration="2.239610234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:39:15.238579789 +0000 UTC m=+1243.332746054" watchObservedRunningTime="2026-09-15 07:39:15.239610234 +0000 UTC m=+1243.333776497"
Sep 15 07:39:21.711357 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:21.711321    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:39:24.046647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:24.046615    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:24.047257 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:24.046664    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:24.059559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:24.059531    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:24.280949 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:24.280924    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:39:43.821605 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:43.821572    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:39:51.808951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.808911    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:39:51.812506 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.812483    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:51.814519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.814499    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-526b4727-serving-cert\""
Sep 15 07:39:51.824003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.823977    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:39:51.928827 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.928787    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:51.929012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.928860    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:51.929012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.928898    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:51.929149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:51.929021    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd9jb\" (UniqueName: \"kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.030485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.030449    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.030657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.030546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd9jb\" (UniqueName: \"kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.030657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.030587    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.030657 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:39:52.030598    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-serving-cert: secret "maas-api-e2e-mt-526b4727-serving-cert" not found
Sep 15 07:39:52.030657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.030627    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.030857 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:39:52.030682    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls podName:3a5c0137-46c4-4e5e-a0e3-41c9871a7aef nodeName:}" failed. No retries permitted until 2026-09-15 07:39:52.530660178 +0000 UTC m=+1280.624826426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls") pod "maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" (UID: "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef") : secret "maas-api-e2e-mt-526b4727-serving-cert" not found
Sep 15 07:39:52.031349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.031332    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.033421 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.033395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.038924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.038898    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd9jb\" (UniqueName: \"kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.535888 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.535851    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.538322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.538298    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") pod \"maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:52.727198 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:52.727163    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:53.008629 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:53.008591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:39:53.009923 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:39:53.009896    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5c0137_46c4_4e5e_a0e3_41c9871a7aef.slice/crio-f267526703af506c3ca86a9be79443b53cc70412c3fb17a5e71f6a9255172f9d WatchSource:0}: Error finding container f267526703af506c3ca86a9be79443b53cc70412c3fb17a5e71f6a9255172f9d: Status 404 returned error can't find the container with id f267526703af506c3ca86a9be79443b53cc70412c3fb17a5e71f6a9255172f9d
Sep 15 07:39:53.434939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:53.434912    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" event={"ID":"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef","Type":"ContainerStarted","Data":"f267526703af506c3ca86a9be79443b53cc70412c3fb17a5e71f6a9255172f9d"}
Sep 15 07:39:54.441396 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:54.441360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" event={"ID":"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef","Type":"ContainerStarted","Data":"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"}
Sep 15 07:39:54.441789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:54.441480    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:39:54.460580 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:39:54.460524    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" podStartSLOduration=3.058923141 podStartE2EDuration="3.460509682s" totalImagesPullingTime="401.586541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:39:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:39:54.45763301 +0000 UTC m=+1282.551799284" watchObservedRunningTime="2026-09-15 07:39:54.460509682 +0000 UTC m=+1282.554675944"
Sep 15 07:40:00.453900 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:00.453869    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:40:06.208738 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:06.208698    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:40:16.009691 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.009657    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:40:16.013597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.013574    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.015903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.015880    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c258f1\"/\"explicit-model-c258f1-kserve-self-signed-certs\""
Sep 15 07:40:16.016585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.016569    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c258f1\"/\"openshift-service-ca.crt\""
Sep 15 07:40:16.016662 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.016621    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c258f1\"/\"kube-root-ca.crt\""
Sep 15 07:40:16.016710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.016673    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c258f1\"/\"default-dockercfg-9rltx\""
Sep 15 07:40:16.024198 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.024173    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:40:16.162046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162008    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.162405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.162569 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162550    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4hk\" (UniqueName: \"kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.162681 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.162808 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162789    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.162930 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.162914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.263855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.263763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.263855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.263800    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.263855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.263830    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264159 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.263956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4hk\" (UniqueName: \"kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264159 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.264001    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.264200    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.264226    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.264350    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.264602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.264582    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.266226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.266200    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.266619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.266602    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.271745 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.271718    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4hk\" (UniqueName: \"kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk\") pod \"explicit-model-c258f1-kserve-6d47f9c55-r7q8s\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") " pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.325046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.325023    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:16.602078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:16.602050    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:40:16.604529 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:40:16.604493    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53555b9a_627f_4dc3_b373_c6ef52696221.slice/crio-2aa57efbb285521c5405ee5d786c7e8cb337d2a1d97825f6af8b45e48912d2c6 WatchSource:0}: Error finding container 2aa57efbb285521c5405ee5d786c7e8cb337d2a1d97825f6af8b45e48912d2c6: Status 404 returned error can't find the container with id 2aa57efbb285521c5405ee5d786c7e8cb337d2a1d97825f6af8b45e48912d2c6
Sep 15 07:40:17.581945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:17.581910    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" event={"ID":"53555b9a-627f-4dc3-b373-c6ef52696221","Type":"ContainerStarted","Data":"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"}
Sep 15 07:40:17.581945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:17.581952    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" event={"ID":"53555b9a-627f-4dc3-b373-c6ef52696221","Type":"ContainerStarted","Data":"2aa57efbb285521c5405ee5d786c7e8cb337d2a1d97825f6af8b45e48912d2c6"}
Sep 15 07:40:17.597843 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:17.597794    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" podStartSLOduration=2.59777947 podStartE2EDuration="2.59777947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:40:17.59634087 +0000 UTC m=+1305.690507133" watchObservedRunningTime="2026-09-15 07:40:17.59777947 +0000 UTC m=+1305.691945732"
Sep 15 07:40:26.325977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:26.325940    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:26.326382 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:26.325989    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:26.338620 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:26.338593    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:26.646762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:26.646679    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:40:36.084581 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.084548    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:40:36.085132 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.084766    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" podUID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" containerName="maas-api" containerID="cri-o://f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7" gracePeriod=30
Sep 15 07:40:36.344364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.344294    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:40:36.439164 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.439132    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") pod \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") "
Sep 15 07:40:36.439333 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.439171    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca\") pod \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") "
Sep 15 07:40:36.439333 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.439232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs\") pod \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") "
Sep 15 07:40:36.439333 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.439256    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd9jb\" (UniqueName: \"kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb\") pod \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\" (UID: \"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef\") "
Sep 15 07:40:36.439648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.439619    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" (UID: "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:40:36.441354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.441324    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" (UID: "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:40:36.441461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.441408    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" (UID: "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:40:36.441461 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.441414    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb" (OuterVolumeSpecName: "kube-api-access-nd9jb") pod "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" (UID: "3a5c0137-46c4-4e5e-a0e3-41c9871a7aef"). InnerVolumeSpecName "kube-api-access-nd9jb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:40:36.540904 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.540871    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:40:36.540904 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.540898    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:40:36.540904 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.540907    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:40:36.541148 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.540916    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd9jb\" (UniqueName: \"kubernetes.io/projected/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef-kube-api-access-nd9jb\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:40:36.697410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.697375    2574 generic.go:356] "Generic (PLEG): container finished" podID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" containerID="f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7" exitCode=0
Sep 15 07:40:36.697560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.697435    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"
Sep 15 07:40:36.697560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.697463    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" event={"ID":"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef","Type":"ContainerDied","Data":"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"}
Sep 15 07:40:36.697560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.697524    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk" event={"ID":"3a5c0137-46c4-4e5e-a0e3-41c9871a7aef","Type":"ContainerDied","Data":"f267526703af506c3ca86a9be79443b53cc70412c3fb17a5e71f6a9255172f9d"}
Sep 15 07:40:36.697560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.697553    2574 scope.go:117] "RemoveContainer" containerID="f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"
Sep 15 07:40:36.707794 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.707777    2574 scope.go:117] "RemoveContainer" containerID="f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"
Sep 15 07:40:36.708057 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:40:36.708039    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7\": container with ID starting with f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7 not found: ID does not exist" containerID="f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"
Sep 15 07:40:36.708095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.708067    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7"} err="failed to get container status \"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7\": rpc error: code = NotFound desc = could not find container \"f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7\": container with ID starting with f9a99412285af8b6b424fc48419823b915797367760a2d22b9f95307453781c7 not found: ID does not exist"
Sep 15 07:40:36.719665 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.719644    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:40:36.723746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:36.723726    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-526b4727-5dd4677ccc-z6xwk"]
Sep 15 07:40:38.379966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:38.379933    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" path="/var/lib/kubelet/pods/3a5c0137-46c4-4e5e-a0e3-41c9871a7aef/volumes"
Sep 15 07:40:39.907153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:39.907103    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:40:54.228375 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:40:54.228339    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:41:00.299491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.299394    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:41:00.299959 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.299653    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" podUID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" containerName="maas-api" containerID="cri-o://7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170" gracePeriod=30
Sep 15 07:41:00.589054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.589023    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:41:00.658348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.658311    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgsd9\" (UniqueName: \"kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9\") pod \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") "
Sep 15 07:41:00.658495 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.658395    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca\") pod \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") "
Sep 15 07:41:00.658495 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.658436    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs\") pod \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") "
Sep 15 07:41:00.658627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.658507    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls\") pod \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\" (UID: \"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81\") "
Sep 15 07:41:00.659182 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.659147    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" (UID: "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:41:00.660920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.660887    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" (UID: "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:41:00.660920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.660895    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9" (OuterVolumeSpecName: "kube-api-access-lgsd9") pod "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" (UID: "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81"). InnerVolumeSpecName "kube-api-access-lgsd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:41:00.661089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.660952    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" (UID: "0635fda3-ad6c-4ef9-994e-ffb02b4c0f81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:41:00.760089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.760049    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:00.760089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.760090    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:00.760303 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.760105    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:00.760303 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.760149    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgsd9\" (UniqueName: \"kubernetes.io/projected/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81-kube-api-access-lgsd9\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:00.843013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.842933    2574 generic.go:356] "Generic (PLEG): container finished" podID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" containerID="7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170" exitCode=0
Sep 15 07:41:00.843013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.842986    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"
Sep 15 07:41:00.843013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.842994    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" event={"ID":"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81","Type":"ContainerDied","Data":"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"}
Sep 15 07:41:00.843323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.843063    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4" event={"ID":"0635fda3-ad6c-4ef9-994e-ffb02b4c0f81","Type":"ContainerDied","Data":"fe647b34c0f42e38395ff2357041b9ae0f4e241455b47ad8e39b523f607ce78f"}
Sep 15 07:41:00.843323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.843096    2574 scope.go:117] "RemoveContainer" containerID="7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"
Sep 15 07:41:00.854626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.854579    2574 scope.go:117] "RemoveContainer" containerID="7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"
Sep 15 07:41:00.854882 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:00.854865    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170\": container with ID starting with 7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170 not found: ID does not exist" containerID="7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"
Sep 15 07:41:00.854947 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.854890    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170"} err="failed to get container status \"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170\": rpc error: code = NotFound desc = could not find container \"7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170\": container with ID starting with 7efda87e30d43afc803e1550aa918a8c3c647397431423b2a406d14829702170 not found: ID does not exist"
Sep 15 07:41:00.867367 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.867346    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:41:00.871196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:00.871170    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-c258f1-5c75748bd-vp7w4"]
Sep 15 07:41:02.221363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:02.221324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:41:02.380409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:02.380366    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" path="/var/lib/kubelet/pods/0635fda3-ad6c-4ef9-994e-ffb02b4c0f81/volumes"
Sep 15 07:41:03.221656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.221617    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:41:03.222296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222271    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" containerName="maas-api"
Sep 15 07:41:03.222369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222299    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" containerName="maas-api"
Sep 15 07:41:03.222369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222333    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" containerName="maas-api"
Sep 15 07:41:03.222369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222342    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" containerName="maas-api"
Sep 15 07:41:03.222526 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222450    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0635fda3-ad6c-4ef9-994e-ffb02b4c0f81" containerName="maas-api"
Sep 15 07:41:03.222526 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.222465    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5c0137-46c4-4e5e-a0e3-41c9871a7aef" containerName="maas-api"
Sep 15 07:41:03.225862 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.225835    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.228696 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.228670    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-3a4c9f-serving-cert\""
Sep 15 07:41:03.238782 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.238712    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:41:03.285225 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.285200    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmz4\" (UniqueName: \"kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.285352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.285241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.285352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.285308    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.285429 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.285394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.386560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.386530    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.386669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.386590    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.386669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.386662    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmz4\" (UniqueName: \"kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.386762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.386690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.386810 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:03.386785    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-serving-cert: secret "maas-api-e2e-shared-a-w2-3a4c9f-serving-cert" not found
Sep 15 07:41:03.386850 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:03.386834    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls podName:818dd830-4abd-44a0-aab0-24a93e2ff59b nodeName:}" failed. No retries permitted until 2026-09-15 07:41:03.886820652 +0000 UTC m=+1351.980986892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls") pod "maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" (UID: "818dd830-4abd-44a0-aab0-24a93e2ff59b") : secret "maas-api-e2e-shared-a-w2-3a4c9f-serving-cert" not found
Sep 15 07:41:03.387340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.387320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.388954 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.388935    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.394488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.394466    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmz4\" (UniqueName: \"kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.891800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.891748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:03.894236 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:03.894210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:04.139157 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:04.139103    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:04.418213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:04.418188    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:41:04.421191 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:41:04.421165    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818dd830_4abd_44a0_aab0_24a93e2ff59b.slice/crio-229ad46a602bc2b6e283f5ac837a79a685c448549f1451738a379aecbe2f6666 WatchSource:0}: Error finding container 229ad46a602bc2b6e283f5ac837a79a685c448549f1451738a379aecbe2f6666: Status 404 returned error can't find the container with id 229ad46a602bc2b6e283f5ac837a79a685c448549f1451738a379aecbe2f6666
Sep 15 07:41:04.422440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:04.422419    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:41:04.874342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:04.874315    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" event={"ID":"818dd830-4abd-44a0-aab0-24a93e2ff59b","Type":"ContainerStarted","Data":"229ad46a602bc2b6e283f5ac837a79a685c448549f1451738a379aecbe2f6666"}
Sep 15 07:41:05.881646 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:05.881610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" event={"ID":"818dd830-4abd-44a0-aab0-24a93e2ff59b","Type":"ContainerStarted","Data":"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"}
Sep 15 07:41:05.882015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:05.881723    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:05.905645 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:05.905599    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" podStartSLOduration=2.480623171 podStartE2EDuration="2.905585642s" totalImagesPullingTime="424.962471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:05.903534432 +0000 UTC m=+1353.997700693" watchObservedRunningTime="2026-09-15 07:41:05.905585642 +0000 UTC m=+1353.999751904"
Sep 15 07:41:09.249892 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.249846    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" podUID="53555b9a-627f-4dc3-b373-c6ef52696221" containerName="main" containerID="cri-o://251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2" gracePeriod=45
Sep 15 07:41:09.502701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.502645    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:41:09.642334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642305    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642341    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642374    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c4hk\" (UniqueName: \"kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642394    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642433    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642511    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs\") pod \"53555b9a-627f-4dc3-b373-c6ef52696221\" (UID: \"53555b9a-627f-4dc3-b373-c6ef52696221\") "
Sep 15 07:41:09.642630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642590    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:09.642779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache" (OuterVolumeSpecName: "model-cache") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:09.642779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642720    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home" (OuterVolumeSpecName: "home") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:09.642882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642855    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.642882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642874    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.642958 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.642885    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.644652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.644620    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk" (OuterVolumeSpecName: "kube-api-access-6c4hk") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "kube-api-access-6c4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:41:09.644652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.644644    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm" (OuterVolumeSpecName: "dshm") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:09.644823 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.644661    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "53555b9a-627f-4dc3-b373-c6ef52696221" (UID: "53555b9a-627f-4dc3-b373-c6ef52696221"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:41:09.743249 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.743226    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53555b9a-627f-4dc3-b373-c6ef52696221-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.743249 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.743245    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/53555b9a-627f-4dc3-b373-c6ef52696221-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.743389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.743254    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4hk\" (UniqueName: \"kubernetes.io/projected/53555b9a-627f-4dc3-b373-c6ef52696221-kube-api-access-6c4hk\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:09.912907 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.912872    2574 generic.go:356] "Generic (PLEG): container finished" podID="53555b9a-627f-4dc3-b373-c6ef52696221" containerID="251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2" exitCode=0
Sep 15 07:41:09.913077 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.912922    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"
Sep 15 07:41:09.913077 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.912950    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" event={"ID":"53555b9a-627f-4dc3-b373-c6ef52696221","Type":"ContainerDied","Data":"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"}
Sep 15 07:41:09.913077 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.912990    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s" event={"ID":"53555b9a-627f-4dc3-b373-c6ef52696221","Type":"ContainerDied","Data":"2aa57efbb285521c5405ee5d786c7e8cb337d2a1d97825f6af8b45e48912d2c6"}
Sep 15 07:41:09.913077 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.913007    2574 scope.go:117] "RemoveContainer" containerID="251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"
Sep 15 07:41:09.924020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.924003    2574 scope.go:117] "RemoveContainer" containerID="251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"
Sep 15 07:41:09.924292 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:09.924274    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2\": container with ID starting with 251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2 not found: ID does not exist" containerID="251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"
Sep 15 07:41:09.924364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.924301    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2"} err="failed to get container status \"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2\": rpc error: code = NotFound desc = could not find container \"251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2\": container with ID starting with 251ad0c707e77290f972ca243eefb6d985097a428b3821109bce13d648acf5d2 not found: ID does not exist"
Sep 15 07:41:09.935295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.935273    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:41:09.937694 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:09.937675    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c258f1/explicit-model-c258f1-kserve-6d47f9c55-r7q8s"]
Sep 15 07:41:10.379976 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:10.379900    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53555b9a-627f-4dc3-b373-c6ef52696221" path="/var/lib/kubelet/pods/53555b9a-627f-4dc3-b373-c6ef52696221/volumes"
Sep 15 07:41:11.893322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:11.893285    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:41:12.126395 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:12.126365    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:41:12.126619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:12.126583    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" containerID="cri-o://21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0" gracePeriod=30
Sep 15 07:41:13.033621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:13.033585    2574 patch_prober.go:28] interesting pod/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 07:41:13.033991 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:13.033638    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:41:17.388368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.388345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:41:17.514536 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514508    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxqn5\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514629    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514654    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514682    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514780    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514812    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514845    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket\") pod \"9e038515-e9e9-42d7-811d-9238cb31b899\" (UID: \"9e038515-e9e9-42d7-811d-9238cb31b899\") "
Sep 15 07:41:17.514894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.514885    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:17.515201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.515156    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:41:17.515201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.515181    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.515201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.515173    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:17.515374 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.515247    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:17.515430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.515374    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:17.516765 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.516742    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:41:17.516886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.516816    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:41:17.517161 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.517130    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:41:17.517161 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.517138    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5" (OuterVolumeSpecName: "kube-api-access-jxqn5") pod "9e038515-e9e9-42d7-811d-9238cb31b899" (UID: "9e038515-e9e9-42d7-811d-9238cb31b899"). InnerVolumeSpecName "kube-api-access-jxqn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:41:17.615951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615928    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxqn5\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-kube-api-access-jxqn5\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.615951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615950    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615959    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e038515-e9e9-42d7-811d-9238cb31b899-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615967    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615975    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e038515-e9e9-42d7-811d-9238cb31b899-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615983    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615990    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e038515-e9e9-42d7-811d-9238cb31b899-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.616087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.615999    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e038515-e9e9-42d7-811d-9238cb31b899-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:41:17.968078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.968050    2574 generic.go:356] "Generic (PLEG): container finished" podID="9e038515-e9e9-42d7-811d-9238cb31b899" containerID="21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0" exitCode=0
Sep 15 07:41:17.968232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.968092    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"
Sep 15 07:41:17.968232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.968137    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerDied","Data":"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"}
Sep 15 07:41:17.968232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.968192    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk" event={"ID":"9e038515-e9e9-42d7-811d-9238cb31b899","Type":"ContainerDied","Data":"0fecdf0019e4cc1c701778f8435ae535aeda15c641839f8e75f7c89139b78b51"}
Sep 15 07:41:17.968232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.968220    2574 scope.go:117] "RemoveContainer" containerID="21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"
Sep 15 07:41:17.977797 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.977779    2574 scope.go:117] "RemoveContainer" containerID="8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"
Sep 15 07:41:17.986201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.986180    2574 scope.go:117] "RemoveContainer" containerID="21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"
Sep 15 07:41:17.986455 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:17.986436    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0\": container with ID starting with 21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0 not found: ID does not exist" containerID="21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"
Sep 15 07:41:17.986496 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.986464    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0"} err="failed to get container status \"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0\": rpc error: code = NotFound desc = could not find container \"21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0\": container with ID starting with 21ef53cb107793f6d66b47f57545a6b0d971f1e1514e77c0b65d585f934a93d0 not found: ID does not exist"
Sep 15 07:41:17.986496 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.986481    2574 scope.go:117] "RemoveContainer" containerID="8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"
Sep 15 07:41:17.986751 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:41:17.986717    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d\": container with ID starting with 8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d not found: ID does not exist" containerID="8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"
Sep 15 07:41:17.986794 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.986760    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d"} err="failed to get container status \"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d\": rpc error: code = NotFound desc = could not find container \"8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d\": container with ID starting with 8ebe88a86dec374e9b036f2f4429686e157482ec213929b97b5504e2dbddb67d not found: ID does not exist"
Sep 15 07:41:17.990217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.990193    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:41:17.999656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:17.999634    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c258f1-openshift-default-576f7b8cb8-wqpvk"]
Sep 15 07:41:18.379530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:18.379448    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" path="/var/lib/kubelet/pods/9e038515-e9e9-42d7-811d-9238cb31b899/volumes"
Sep 15 07:41:29.954218 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954175    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:41:29.954920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954897    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.955005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954923    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.955005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954936    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53555b9a-627f-4dc3-b373-c6ef52696221" containerName="main"
Sep 15 07:41:29.955005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954944    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="53555b9a-627f-4dc3-b373-c6ef52696221" containerName="main"
Sep 15 07:41:29.955005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954963    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.955005 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.954968    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.955270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.955041    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="53555b9a-627f-4dc3-b373-c6ef52696221" containerName="main"
Sep 15 07:41:29.955270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.955050    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.955270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.955056    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e038515-e9e9-42d7-811d-9238cb31b899" containerName="istio-proxy"
Sep 15 07:41:29.959829 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.959807    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:29.962160 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.962137    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-27b9d8-openshift-default-dockercfg-gtscz\""
Sep 15 07:41:29.970020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:29.969997    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:41:30.128830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.128794    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.128855    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.128926    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.128976    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.129011    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.129034    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.129052    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx4k\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.129075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.129255 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.129153    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230377 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230305    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230377 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230472    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230531    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230566    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230593    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx4k\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230631    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230660    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230673    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.230971 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230916    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.231013 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.230986    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.231145 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.231108    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.231353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.231331    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.233235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.233214    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.233566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.233543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.238922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.238897    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.239047 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.239027    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx4k\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k\") pod \"e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") " pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.271944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.271921    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:30.550288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.548040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:41:30.553216 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:41:30.553174    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9693864f_0777_449a_b4ca_65dee8c3ced6.slice/crio-78393e1459bd501e720f4aeb47ac19a4838849642eb1b7d673658ab022166730 WatchSource:0}: Error finding container 78393e1459bd501e720f4aeb47ac19a4838849642eb1b7d673658ab022166730: Status 404 returned error can't find the container with id 78393e1459bd501e720f4aeb47ac19a4838849642eb1b7d673658ab022166730
Sep 15 07:41:30.555482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.555449    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:41:30.555594 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.555549    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:41:30.555653 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:30.555595    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:41:31.045932 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:31.045902    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerStarted","Data":"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"}
Sep 15 07:41:31.046411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:31.045953    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerStarted","Data":"78393e1459bd501e720f4aeb47ac19a4838849642eb1b7d673658ab022166730"}
Sep 15 07:41:31.067322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:31.067271    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podStartSLOduration=2.067257723 podStartE2EDuration="2.067257723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:31.064277715 +0000 UTC m=+1379.158443980" watchObservedRunningTime="2026-09-15 07:41:31.067257723 +0000 UTC m=+1379.161423985"
Sep 15 07:41:31.272579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:31.272546    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:31.277719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:31.277687    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:32.052187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:32.052152    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:32.053241 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:32.053220    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:41:34.149254 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.149217    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:41:34.154651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.154629    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.156888 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.156866    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"distinct-w0-2735c5-kserve-self-signed-certs\""
Sep 15 07:41:34.163021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.162999    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:41:34.267380 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267347    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhh57\" (UniqueName: \"kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.267547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.267547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267413    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.267547 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267528    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.267748 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.267748 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.267648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368096 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhh57\" (UniqueName: \"kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368107    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368162    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368270 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368222    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368608 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368583    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368694 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368621    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.368694 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.368680    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.370552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.370522    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.370717 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.370701    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.376848 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.376820    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhh57\" (UniqueName: \"kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57\") pod \"distinct-w0-2735c5-kserve-64677d94f6-hsjbp\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") " pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.463887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.463858    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:34.739719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:34.739643    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:41:34.742104 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:41:34.742070    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b8b0a55_5c0a_47b1_a38c_76a414c1c05b.slice/crio-d8d2e4c05c9506b200a292cb421372b5447b6b910886d292bec25608f409a0a9 WatchSource:0}: Error finding container d8d2e4c05c9506b200a292cb421372b5447b6b910886d292bec25608f409a0a9: Status 404 returned error can't find the container with id d8d2e4c05c9506b200a292cb421372b5447b6b910886d292bec25608f409a0a9
Sep 15 07:41:35.071599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:35.071508    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" event={"ID":"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b","Type":"ContainerStarted","Data":"7f529d43622faadccb1e9e633ae9856f9f910214648b609e430ddf43d9a4a919"}
Sep 15 07:41:35.071599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:35.071562    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" event={"ID":"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b","Type":"ContainerStarted","Data":"d8d2e4c05c9506b200a292cb421372b5447b6b910886d292bec25608f409a0a9"}
Sep 15 07:41:35.091791 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:35.091742    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" podStartSLOduration=1.091729149 podStartE2EDuration="1.091729149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:35.09043431 +0000 UTC m=+1383.184600572" watchObservedRunningTime="2026-09-15 07:41:35.091729149 +0000 UTC m=+1383.185895410"
Sep 15 07:41:36.113266 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:36.113231    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:41:37.127607 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.127564    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:41:37.131319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.131299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.134004 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.133978    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-27b9d8-serving-cert\""
Sep 15 07:41:37.142916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.142895    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:41:37.195433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.195405    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.195566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.195462    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.195609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.195559    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.195609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.195590    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfzzc\" (UniqueName: \"kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.296602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.296563    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.296762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.296678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.296762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.296714    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfzzc\" (UniqueName: \"kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.296868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.296785    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.297183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.297159    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.299185 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.299160    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.299285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.299267    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.303994 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.303971    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfzzc\" (UniqueName: \"kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc\") pod \"maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.444335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.444312    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:37.716878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:37.716850    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:41:37.718084 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:41:37.718062    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82cf8264_b253_4ff9_867f_9391f08f09d0.slice/crio-6c1874521b39c3f49e358beb697eeda092c9db050351eff89eec890869f268b8 WatchSource:0}: Error finding container 6c1874521b39c3f49e358beb697eeda092c9db050351eff89eec890869f268b8: Status 404 returned error can't find the container with id 6c1874521b39c3f49e358beb697eeda092c9db050351eff89eec890869f268b8
Sep 15 07:41:38.092058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:38.091978    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" event={"ID":"82cf8264-b253-4ff9-867f-9391f08f09d0","Type":"ContainerStarted","Data":"6c1874521b39c3f49e358beb697eeda092c9db050351eff89eec890869f268b8"}
Sep 15 07:41:39.099550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:39.099515    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" event={"ID":"82cf8264-b253-4ff9-867f-9391f08f09d0","Type":"ContainerStarted","Data":"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"}
Sep 15 07:41:39.099964 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:39.099693    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:39.118110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:39.118065    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" podStartSLOduration=1.287732785 podStartE2EDuration="2.118050881s" totalImagesPullingTime="830.318096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:41:39.115080992 +0000 UTC m=+1387.209247257" watchObservedRunningTime="2026-09-15 07:41:39.118050881 +0000 UTC m=+1387.212217143"
Sep 15 07:41:44.464596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:44.464566    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:44.465035 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:44.464614    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:44.476861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:44.476836    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:41:45.111126 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:45.111083    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:41:45.156784 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:41:45.156756    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:42:03.094277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.094238    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 07:42:03.094762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.094338    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 07:42:03.763298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.763263    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-600778"]
Sep 15 07:42:03.766772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.766747    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:03.768840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.768816    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8zjf\""
Sep 15 07:42:03.772434 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.772410    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-600778"]
Sep 15 07:42:03.854100 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.854067    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tcq5\" (UniqueName: \"kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5\") pod \"test-curl-7657-600778\" (UID: \"d7ef07af-b122-450a-ab72-061dbcae87c2\") " pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:03.955134 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.955079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tcq5\" (UniqueName: \"kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5\") pod \"test-curl-7657-600778\" (UID: \"d7ef07af-b122-450a-ab72-061dbcae87c2\") " pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:03.963574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:03.963540    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tcq5\" (UniqueName: \"kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5\") pod \"test-curl-7657-600778\" (UID: \"d7ef07af-b122-450a-ab72-061dbcae87c2\") " pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:04.079800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.079726    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:04.254437 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.254403    2574 generic.go:356] "Generic (PLEG): container finished" podID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerID="eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11" exitCode=0
Sep 15 07:42:04.254760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.254454    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerDied","Data":"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"}
Sep 15 07:42:04.254977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.254959    2574 scope.go:117] "RemoveContainer" containerID="eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"
Sep 15 07:42:04.256337 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.256303    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:04.256441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.256394    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:04.256491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.256475    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:04.354108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:04.354071    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-600778"]
Sep 15 07:42:04.355214 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:04.355189    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd7ef07af_b122_450a_ab72_061dbcae87c2.slice/crio-2f3fb1ed7a6438e5a8fb17cb123a0a405b1e8ccb4fa3ee9e6e6c8f7d215567e2 WatchSource:0}: Error finding container 2f3fb1ed7a6438e5a8fb17cb123a0a405b1e8ccb4fa3ee9e6e6c8f7d215567e2: Status 404 returned error can't find the container with id 2f3fb1ed7a6438e5a8fb17cb123a0a405b1e8ccb4fa3ee9e6e6c8f7d215567e2
Sep 15 07:42:05.266136 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.266071    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-600778" event={"ID":"d7ef07af-b122-450a-ab72-061dbcae87c2","Type":"ContainerStarted","Data":"d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e"}
Sep 15 07:42:05.266626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.266144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-600778" event={"ID":"d7ef07af-b122-450a-ab72-061dbcae87c2","Type":"ContainerStarted","Data":"2f3fb1ed7a6438e5a8fb17cb123a0a405b1e8ccb4fa3ee9e6e6c8f7d215567e2"}
Sep 15 07:42:05.268607 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.268568    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerStarted","Data":"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"}
Sep 15 07:42:05.272684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.272652    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:42:05.272991 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.272964    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 07:42:05.273061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.273034    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 07:42:05.284074 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.284031    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7657-600778" podStartSLOduration=2.284017858 podStartE2EDuration="2.284017858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:05.283580496 +0000 UTC m=+1413.377746759" watchObservedRunningTime="2026-09-15 07:42:05.284017858 +0000 UTC m=+1413.378184121"
Sep 15 07:42:05.917062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.917027    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-600778"]
Sep 15 07:42:05.921849 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:05.921823    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-600778"]
Sep 15 07:42:06.042710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.042637    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-b30b84"]
Sep 15 07:42:06.043058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.043046    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" containerName="test-curl-7657-600778"
Sep 15 07:42:06.043104 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.043060    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" containerName="test-curl-7657-600778"
Sep 15 07:42:06.043163 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.043142    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" containerName="test-curl-7657-600778"
Sep 15 07:42:06.045509 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.045490    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:06.071139 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.063864    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-b30b84"]
Sep 15 07:42:06.180641 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.180607    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rrph\" (UniqueName: \"kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph\") pod \"test-curl-7657-b30b84\" (UID: \"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4\") " pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:06.275595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.275558    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 07:42:06.276058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.275618    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 07:42:06.278732 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:06.278710    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7657-600778" container="test-curl-7657-600778"
Sep 15 07:42:06.281876 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.281852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rrph\" (UniqueName: \"kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph\") pod \"test-curl-7657-b30b84\" (UID: \"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4\") " pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:06.292321 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.292302    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rrph\" (UniqueName: \"kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph\") pod \"test-curl-7657-b30b84\" (UID: \"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4\") " pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:06.392924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.392865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:06.676024 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:06.675991    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-b30b84"]
Sep 15 07:42:06.679268 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:06.679232    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0066023c_6b9a_46f7_baa3_7e5f9ce8b8b4.slice/crio-fa2af17ba537c3ac1398539167eafe8dad7c3cfcaa4173f70e8686016695503d WatchSource:0}: Error finding container fa2af17ba537c3ac1398539167eafe8dad7c3cfcaa4173f70e8686016695503d: Status 404 returned error can't find the container with id fa2af17ba537c3ac1398539167eafe8dad7c3cfcaa4173f70e8686016695503d
Sep 15 07:42:07.273359 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.272945    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 07:42:07.273359 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.273004    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 07:42:07.287246 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.287216    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-b30b84" event={"ID":"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4","Type":"ContainerStarted","Data":"8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18"}
Sep 15 07:42:07.287661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.287254    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-b30b84" event={"ID":"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4","Type":"ContainerStarted","Data":"fa2af17ba537c3ac1398539167eafe8dad7c3cfcaa4173f70e8686016695503d"}
Sep 15 07:42:07.287661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.287295    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7657-600778" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" containerName="test-curl-7657-600778" containerID="cri-o://d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e" gracePeriod=2
Sep 15 07:42:07.302213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.302171    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7657-b30b84" podStartSLOduration=1.3021559360000001 podStartE2EDuration="1.302155936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:07.299658068 +0000 UTC m=+1415.393824340" watchObservedRunningTime="2026-09-15 07:42:07.302155936 +0000 UTC m=+1415.396322197"
Sep 15 07:42:07.936566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.936526    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-b30b84"]
Sep 15 07:42:07.940762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:07.940736    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-b30b84"]
Sep 15 07:42:08.067353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.067245    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-fd454b"]
Sep 15 07:42:08.068183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.068155    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" containerName="test-curl-7657-b30b84"
Sep 15 07:42:08.068351 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.068338    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" containerName="test-curl-7657-b30b84"
Sep 15 07:42:08.068575 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.068561    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" containerName="test-curl-7657-b30b84"
Sep 15 07:42:08.079280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.079258    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-fd454b"]
Sep 15 07:42:08.079502 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.079488    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:08.203914 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.203887    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l95ss\" (UniqueName: \"kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss\") pod \"test-curl-7657-fd454b\" (UID: \"0043239b-7068-4cad-8ad5-d81d53b759cb\") " pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:08.273204 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.273174    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 07:42:08.273427 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.273402    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 07:42:08.293245 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:08.293218    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7657-b30b84" container="test-curl-7657-b30b84"
Sep 15 07:42:08.305226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.305206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l95ss\" (UniqueName: \"kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss\") pod \"test-curl-7657-fd454b\" (UID: \"0043239b-7068-4cad-8ad5-d81d53b759cb\") " pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:08.313309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.313287    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l95ss\" (UniqueName: \"kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss\") pod \"test-curl-7657-fd454b\" (UID: \"0043239b-7068-4cad-8ad5-d81d53b759cb\") " pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:08.393944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.393871    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:08.677054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:08.677018    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-fd454b"]
Sep 15 07:42:08.680901 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:08.680871    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0043239b_7068_4cad_8ad5_d81d53b759cb.slice/crio-9a9984903ae9c37001c8a1f171a9968268bf02eb5f49c6c8d12e914646e8aa2e WatchSource:0}: Error finding container 9a9984903ae9c37001c8a1f171a9968268bf02eb5f49c6c8d12e914646e8aa2e: Status 404 returned error can't find the container with id 9a9984903ae9c37001c8a1f171a9968268bf02eb5f49c6c8d12e914646e8aa2e
Sep 15 07:42:09.273104 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.273063    2574 patch_prober.go:28] interesting pod/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Sep 15 07:42:09.273411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.273376    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Sep 15 07:42:09.302596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.302565    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-fd454b" event={"ID":"0043239b-7068-4cad-8ad5-d81d53b759cb","Type":"ContainerStarted","Data":"b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba"}
Sep 15 07:42:09.302922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.302597    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7657-b30b84" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" containerName="test-curl-7657-b30b84" containerID="cri-o://8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18" gracePeriod=2
Sep 15 07:42:09.302922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.302611    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-fd454b" event={"ID":"0043239b-7068-4cad-8ad5-d81d53b759cb","Type":"ContainerStarted","Data":"9a9984903ae9c37001c8a1f171a9968268bf02eb5f49c6c8d12e914646e8aa2e"}
Sep 15 07:42:09.318786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.318744    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7657-fd454b" podStartSLOduration=1.318732389 podStartE2EDuration="1.318732389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:09.315096802 +0000 UTC m=+1417.409263065" watchObservedRunningTime="2026-09-15 07:42:09.318732389 +0000 UTC m=+1417.412898650"
Sep 15 07:42:09.516032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.516004    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:09.618729 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.618651    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tcq5\" (UniqueName: \"kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5\") pod \"d7ef07af-b122-450a-ab72-061dbcae87c2\" (UID: \"d7ef07af-b122-450a-ab72-061dbcae87c2\") "
Sep 15 07:42:09.620524 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.620501    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5" (OuterVolumeSpecName: "kube-api-access-2tcq5") pod "d7ef07af-b122-450a-ab72-061dbcae87c2" (UID: "d7ef07af-b122-450a-ab72-061dbcae87c2"). InnerVolumeSpecName "kube-api-access-2tcq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:09.719812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.719783    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tcq5\" (UniqueName: \"kubernetes.io/projected/d7ef07af-b122-450a-ab72-061dbcae87c2-kube-api-access-2tcq5\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:09.937961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.937927    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-fd454b"]
Sep 15 07:42:09.941869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:09.941840    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-fd454b"]
Sep 15 07:42:10.069502 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.069470    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7657-7b55b6"]
Sep 15 07:42:10.070371 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.070345    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" containerName="test-curl-7657-fd454b"
Sep 15 07:42:10.070560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.070545    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" containerName="test-curl-7657-fd454b"
Sep 15 07:42:10.070766 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.070754    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" containerName="test-curl-7657-fd454b"
Sep 15 07:42:10.074261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.074236    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-7b55b6"]
Sep 15 07:42:10.074490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.074341    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:10.223907 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.223812    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwpv\" (UniqueName: \"kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv\") pod \"test-curl-7657-7b55b6\" (UID: \"07beba14-074b-4d4e-94d0-84f258900df8\") " pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:10.277906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.277873    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:42:10.278285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.278260    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:42:10.279081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.279061    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:42:10.309493 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.309467    2574 generic.go:356] "Generic (PLEG): container finished" podID="d7ef07af-b122-450a-ab72-061dbcae87c2" containerID="d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e" exitCode=137
Sep 15 07:42:10.309845 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.309522    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:10.309845 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.309560    2574 scope.go:117] "RemoveContainer" containerID="d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e"
Sep 15 07:42:10.309845 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:10.309736    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7657-fd454b" container="test-curl-7657-fd454b"
Sep 15 07:42:10.319775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.319760    2574 scope.go:117] "RemoveContainer" containerID="d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e"
Sep 15 07:42:10.320082 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:10.320063    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e\": container with ID starting with d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e not found: ID does not exist" containerID="d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e"
Sep 15 07:42:10.320184 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.320093    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e"} err="failed to get container status \"d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e\": rpc error: code = NotFound desc = could not find container \"d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e\": container with ID starting with d3d2a6d0e7027279b5a4f1fca40248976d5a9bc54a3830ede7afc731693f545e not found: ID does not exist"
Sep 15 07:42:10.320243 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:10.320219    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7657-600778\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" pod="opendatahub/test-curl-7657-600778"
Sep 15 07:42:10.324654 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.324624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwpv\" (UniqueName: \"kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv\") pod \"test-curl-7657-7b55b6\" (UID: \"07beba14-074b-4d4e-94d0-84f258900df8\") " pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:10.331912 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.331890    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwpv\" (UniqueName: \"kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv\") pod \"test-curl-7657-7b55b6\" (UID: \"07beba14-074b-4d4e-94d0-84f258900df8\") " pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:10.378906 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.378879    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ef07af-b122-450a-ab72-061dbcae87c2" path="/var/lib/kubelet/pods/d7ef07af-b122-450a-ab72-061dbcae87c2/volumes"
Sep 15 07:42:10.385801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.385783    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:10.666795 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:10.666770    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7657-7b55b6"]
Sep 15 07:42:10.668100 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:10.668073    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07beba14_074b_4d4e_94d0_84f258900df8.slice/crio-fa100f9eb637b3af14aa20f2b80c563c8b0618ef55abdc46aa12f7783e1b2b4c WatchSource:0}: Error finding container fa100f9eb637b3af14aa20f2b80c563c8b0618ef55abdc46aa12f7783e1b2b4c: Status 404 returned error can't find the container with id fa100f9eb637b3af14aa20f2b80c563c8b0618ef55abdc46aa12f7783e1b2b4c
Sep 15 07:42:11.318695 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.318665    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-7b55b6" event={"ID":"07beba14-074b-4d4e-94d0-84f258900df8","Type":"ContainerStarted","Data":"6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3"}
Sep 15 07:42:11.318997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.318714    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7657-7b55b6" event={"ID":"07beba14-074b-4d4e-94d0-84f258900df8","Type":"ContainerStarted","Data":"fa100f9eb637b3af14aa20f2b80c563c8b0618ef55abdc46aa12f7783e1b2b4c"}
Sep 15 07:42:11.319924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.319895    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7657-fd454b" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" containerName="test-curl-7657-fd454b" containerID="cri-o://b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba" gracePeriod=2
Sep 15 07:42:11.331796 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.331757    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7657-7b55b6" podStartSLOduration=1.33174226 podStartE2EDuration="1.33174226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:11.331226632 +0000 UTC m=+1419.425392890" watchObservedRunningTime="2026-09-15 07:42:11.33174226 +0000 UTC m=+1419.425908540"
Sep 15 07:42:11.490774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.490753    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:11.536545 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.536525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rrph\" (UniqueName: \"kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph\") pod \"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4\" (UID: \"0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4\") "
Sep 15 07:42:11.538376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.538352    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph" (OuterVolumeSpecName: "kube-api-access-4rrph") pod "0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" (UID: "0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4"). InnerVolumeSpecName "kube-api-access-4rrph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:11.637355 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.637304    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rrph\" (UniqueName: \"kubernetes.io/projected/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4-kube-api-access-4rrph\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:11.902025 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.901960    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7657-7b55b6"]
Sep 15 07:42:11.907347 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:11.907320    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7657-7b55b6"]
Sep 15 07:42:12.327097 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.327063    2574 generic.go:356] "Generic (PLEG): container finished" podID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" containerID="8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18" exitCode=137
Sep 15 07:42:12.327552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.327108    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:12.327552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.327164    2574 scope.go:117] "RemoveContainer" containerID="8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18"
Sep 15 07:42:12.327552 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:12.327543    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7657-7b55b6" container="test-curl-7657-7b55b6"
Sep 15 07:42:12.338273 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:12.338244    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7657-b30b84\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:12.339801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.339785    2574 scope.go:117] "RemoveContainer" containerID="8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18"
Sep 15 07:42:12.340063 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:12.340048    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18\": container with ID starting with 8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18 not found: ID does not exist" containerID="8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18"
Sep 15 07:42:12.340106 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.340070    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18"} err="failed to get container status \"8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18\": rpc error: code = NotFound desc = could not find container \"8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18\": container with ID starting with 8a4ddc1f6bf85f2acbf6acd36f6cd04ba0e55477924bbb8b74862c5569de3b18 not found: ID does not exist"
Sep 15 07:42:12.378374 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:12.378344    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7657-b30b84\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" pod="opendatahub/test-curl-7657-b30b84"
Sep 15 07:42:12.379693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.379673    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4" path="/var/lib/kubelet/pods/0066023c-6b9a-46f7-baa3-7e5f9ce8b8b4/volumes"
Sep 15 07:42:12.741004 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.740970    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:42:12.741490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.741473    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07beba14-074b-4d4e-94d0-84f258900df8" containerName="test-curl-7657-7b55b6"
Sep 15 07:42:12.741549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.741491    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07beba14-074b-4d4e-94d0-84f258900df8" containerName="test-curl-7657-7b55b6"
Sep 15 07:42:12.741600 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.741591    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07beba14-074b-4d4e-94d0-84f258900df8" containerName="test-curl-7657-7b55b6"
Sep 15 07:42:12.744774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.744757    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.746868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.746846    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b21d1ff2-openshift-default-dockercfg-k7zht\""
Sep 15 07:42:12.757479 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.757330    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:42:12.845585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845556    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrqr\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.845768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845615    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.845768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845685    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.845768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845727    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.845768 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845753    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.846008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845862    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.846008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845886    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.846008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845902    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.846008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.845918    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946635 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946635 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946639    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946859 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946657    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946859 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946859 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946853    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrqr\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946998 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.946998 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.946986    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.947110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.947042    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.947110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.947075    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.947110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.947097    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.947330 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.947154    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.947445 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.947420    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.951172 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.948395    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.951669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.951563    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.952653 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.951835    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.954644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.954618    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.963558 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.963529    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:12.965000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:12.964972    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrqr\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr\") pod \"e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") " pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:13.060462 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.060406    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:13.334368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.334341    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7657-7b55b6" podUID="07beba14-074b-4d4e-94d0-84f258900df8" containerName="test-curl-7657-7b55b6" containerID="cri-o://6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3" gracePeriod=2
Sep 15 07:42:13.339870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.339843    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:42:13.379094 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:13.379065    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dab330_9a2f_48e0_ae9f_6abe663b5a69.slice/crio-9645311b6b4b4d6a8f861cb01198781ad9eed37008ebda650bc350e180648e72 WatchSource:0}: Error finding container 9645311b6b4b4d6a8f861cb01198781ad9eed37008ebda650bc350e180648e72: Status 404 returned error can't find the container with id 9645311b6b4b4d6a8f861cb01198781ad9eed37008ebda650bc350e180648e72
Sep 15 07:42:13.381616 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.381578    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:13.381688 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.381669    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:13.381727 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.381707    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:13.477924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.477899    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:13.553274 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.553243    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l95ss\" (UniqueName: \"kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss\") pod \"0043239b-7068-4cad-8ad5-d81d53b759cb\" (UID: \"0043239b-7068-4cad-8ad5-d81d53b759cb\") "
Sep 15 07:42:13.555362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.555336    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss" (OuterVolumeSpecName: "kube-api-access-l95ss") pod "0043239b-7068-4cad-8ad5-d81d53b759cb" (UID: "0043239b-7068-4cad-8ad5-d81d53b759cb"). InnerVolumeSpecName "kube-api-access-l95ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:13.654835 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.654764    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l95ss\" (UniqueName: \"kubernetes.io/projected/0043239b-7068-4cad-8ad5-d81d53b759cb-kube-api-access-l95ss\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:13.715943 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:13.715917    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:42:14.342111 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.342022    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerStarted","Data":"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"}
Sep 15 07:42:14.342111 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.342082    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerStarted","Data":"9645311b6b4b4d6a8f861cb01198781ad9eed37008ebda650bc350e180648e72"}
Sep 15 07:42:14.343441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.343411    2574 generic.go:356] "Generic (PLEG): container finished" podID="0043239b-7068-4cad-8ad5-d81d53b759cb" containerID="b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba" exitCode=137
Sep 15 07:42:14.343572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.343448    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:14.343572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.343470    2574 scope.go:117] "RemoveContainer" containerID="b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba"
Sep 15 07:42:14.353560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.353542    2574 scope.go:117] "RemoveContainer" containerID="b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba"
Sep 15 07:42:14.353808 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:14.353788    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba\": container with ID starting with b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba not found: ID does not exist" containerID="b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba"
Sep 15 07:42:14.353871 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.353816    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba"} err="failed to get container status \"b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba\": rpc error: code = NotFound desc = could not find container \"b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba\": container with ID starting with b326f4a0d9c7e6dbe2f0811b1b83735e9adebd50204ae428fb6c5b96ca00d5ba not found: ID does not exist"
Sep 15 07:42:14.362429 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:14.362397    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7657-fd454b\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" pod="opendatahub/test-curl-7657-fd454b"
Sep 15 07:42:14.363860 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.363822    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podStartSLOduration=2.36381032 podStartE2EDuration="2.36381032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:14.360753582 +0000 UTC m=+1422.454919856" watchObservedRunningTime="2026-09-15 07:42:14.36381032 +0000 UTC m=+1422.457976583"
Sep 15 07:42:14.379296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:14.379272    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0043239b-7068-4cad-8ad5-d81d53b759cb" path="/var/lib/kubelet/pods/0043239b-7068-4cad-8ad5-d81d53b759cb/volumes"
Sep 15 07:42:15.061444 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.061403    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:15.066403 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.066377    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:15.353181 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.353152    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:15.354362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.354344    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:15.528339 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.528314    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:15.574807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.574781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwwpv\" (UniqueName: \"kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv\") pod \"07beba14-074b-4d4e-94d0-84f258900df8\" (UID: \"07beba14-074b-4d4e-94d0-84f258900df8\") "
Sep 15 07:42:15.576921 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.576889    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv" (OuterVolumeSpecName: "kube-api-access-jwwpv") pod "07beba14-074b-4d4e-94d0-84f258900df8" (UID: "07beba14-074b-4d4e-94d0-84f258900df8"). InnerVolumeSpecName "kube-api-access-jwwpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:15.676465 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:15.676443    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwpv\" (UniqueName: \"kubernetes.io/projected/07beba14-074b-4d4e-94d0-84f258900df8-kube-api-access-jwwpv\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:16.368137 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.364692    2574 generic.go:356] "Generic (PLEG): container finished" podID="07beba14-074b-4d4e-94d0-84f258900df8" containerID="6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3" exitCode=137
Sep 15 07:42:16.368137 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.365693    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:16.374343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.374313    2574 scope.go:117] "RemoveContainer" containerID="6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3"
Sep 15 07:42:16.385405 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:16.384787    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7657-7b55b6\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="07beba14-074b-4d4e-94d0-84f258900df8" pod="opendatahub/test-curl-7657-7b55b6"
Sep 15 07:42:16.391448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.391418    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07beba14-074b-4d4e-94d0-84f258900df8" path="/var/lib/kubelet/pods/07beba14-074b-4d4e-94d0-84f258900df8/volumes"
Sep 15 07:42:16.392073 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.391974    2574 scope.go:117] "RemoveContainer" containerID="6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3"
Sep 15 07:42:16.392339 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:16.392310    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3\": container with ID starting with 6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3 not found: ID does not exist" containerID="6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3"
Sep 15 07:42:16.392399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:16.392347    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3"} err="failed to get container status \"6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3\": rpc error: code = NotFound desc = could not find container \"6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3\": container with ID starting with 6a89e1f279fac505be933373b64adca345aa4d82a07e95a32d1cbdf73f2985e3 not found: ID does not exist"
Sep 15 07:42:20.908884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:20.908707    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:42:20.912032 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:20.912010    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:20.914069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:20.914039    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b21d1ff2-serving-cert\""
Sep 15 07:42:20.922987 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:20.922964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:42:21.030253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.030212    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.030430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.030262    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.030430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.030285    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zcj\" (UniqueName: \"kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.030430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.030408    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.131525 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.131492    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.131698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.131567    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.131698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.131592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.131698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.131606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zcj\" (UniqueName: \"kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.131876 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:21.131725    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-serving-cert: secret "maas-api-e2e-mt-b21d1ff2-serving-cert" not found
Sep 15 07:42:21.131876 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:21.131803    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls podName:d00749b8-f119-47d3-beee-85091b312d52 nodeName:}" failed. No retries permitted until 2026-09-15 07:42:21.631781899 +0000 UTC m=+1429.725948160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls") pod "maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" (UID: "d00749b8-f119-47d3-beee-85091b312d52") : secret "maas-api-e2e-mt-b21d1ff2-serving-cert" not found
Sep 15 07:42:21.132252 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.132232    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.134003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.133988    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.141020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.141000    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zcj\" (UniqueName: \"kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.636553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.636506    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.640080 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.639963    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") pod \"maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:21.824441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:21.824409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:22.106795 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:22.106768    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:42:22.107533 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:42:22.107511    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd00749b8_f119_47d3_beee_85091b312d52.slice/crio-7886959abe35d7aa7715b995f699a7842769791496727efcf162f2af71c54251 WatchSource:0}: Error finding container 7886959abe35d7aa7715b995f699a7842769791496727efcf162f2af71c54251: Status 404 returned error can't find the container with id 7886959abe35d7aa7715b995f699a7842769791496727efcf162f2af71c54251
Sep 15 07:42:22.403357 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:22.403276    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" event={"ID":"d00749b8-f119-47d3-beee-85091b312d52","Type":"ContainerStarted","Data":"7886959abe35d7aa7715b995f699a7842769791496727efcf162f2af71c54251"}
Sep 15 07:42:24.415688 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.415649    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" event={"ID":"d00749b8-f119-47d3-beee-85091b312d52","Type":"ContainerStarted","Data":"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"}
Sep 15 07:42:24.416093 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.415797    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:24.434607 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.434559    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" podStartSLOduration=2.746496636 podStartE2EDuration="4.434544285s" totalImagesPullingTime="1.688047649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:24.432757084 +0000 UTC m=+1432.526923345" watchObservedRunningTime="2026-09-15 07:42:24.434544285 +0000 UTC m=+1432.528710542"
Sep 15 07:42:24.787644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.787548    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:42:24.790289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.790274    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.792480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.792456    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c3abd1\"/\"openshift-service-ca.crt\""
Sep 15 07:42:24.792572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.792499    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-c3abd1\"/\"kube-root-ca.crt\""
Sep 15 07:42:24.792572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.792557    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c3abd1\"/\"default-dockercfg-pj77l\""
Sep 15 07:42:24.792727 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.792713    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-c3abd1\"/\"noten-model-c3abd1-kserve-self-signed-certs\""
Sep 15 07:42:24.799640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.799611    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:42:24.869075 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.869224 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869105    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.869293 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.869293 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869245    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrt2\" (UniqueName: \"kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.869400 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869336    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.869451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.869396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970535 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970438    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970535 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970518    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970754 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970754 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970597    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970754 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970679    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970754 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970706    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrt2\" (UniqueName: \"kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970875    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970888    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.970977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.970954    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.973294 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.973258    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.973441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.973423    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:24.978067 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:24.978043    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrt2\" (UniqueName: \"kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2\") pod \"noten-model-c3abd1-kserve-58857fbbff-ccc6p\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") " pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:25.100707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:25.100628    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:25.380912 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:25.380865    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:42:25.423585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:25.423557    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" event={"ID":"beb1710f-2c29-4713-b5d0-050da3bbee64","Type":"ContainerStarted","Data":"aa22278d49cda1311f2f8060d00084136c3784130e1a8f3e35c9f2f63d40fe5a"}
Sep 15 07:42:26.430516 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:26.430477    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" event={"ID":"beb1710f-2c29-4713-b5d0-050da3bbee64","Type":"ContainerStarted","Data":"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"}
Sep 15 07:42:26.450482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:26.450422    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" podStartSLOduration=2.45040127 podStartE2EDuration="2.45040127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:42:26.446344987 +0000 UTC m=+1434.540511250" watchObservedRunningTime="2026-09-15 07:42:26.45040127 +0000 UTC m=+1434.544567533"
Sep 15 07:42:30.428786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:30.428758    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:42:35.101354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:35.101315    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:35.101811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:35.101457    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:35.115526 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:35.115502    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:35.503816 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:35.503787    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:42:37.407565 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:37.407535    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:42:46.396285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:46.396245    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 15 07:42:46.396822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:46.396319    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 07:42:48.577878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.577842    2574 generic.go:356] "Generic (PLEG): container finished" podID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerID="5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0" exitCode=0
Sep 15 07:42:48.578405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.577919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerDied","Data":"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"}
Sep 15 07:42:48.578473 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.578446    2574 scope.go:117] "RemoveContainer" containerID="5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"
Sep 15 07:42:48.579636 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.579608    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:48.579719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.579678    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:48.579719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:48.579711    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:42:49.593067 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:49.593018    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerStarted","Data":"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"}
Sep 15 07:42:50.061719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:50.061681    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:50.062106 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:50.062011    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:50.062370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:50.062345    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:50.788108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:50.788064    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:42:51.060961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:51.060863    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:51.060961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:51.060940    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:51.630046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:51.630013    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:42:51.630406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:51.630376    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" podUID="dded2457-2a7c-47f5-b50a-17f672a35c8e" containerName="maas-api" containerID="cri-o://7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d" gracePeriod=30
Sep 15 07:42:51.978175 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:51.977446    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:42:52.031414 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.031383    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca\") pod \"dded2457-2a7c-47f5-b50a-17f672a35c8e\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") "
Sep 15 07:42:52.031595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.031475    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls\") pod \"dded2457-2a7c-47f5-b50a-17f672a35c8e\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") "
Sep 15 07:42:52.031595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.031513    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnppf\" (UniqueName: \"kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf\") pod \"dded2457-2a7c-47f5-b50a-17f672a35c8e\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") "
Sep 15 07:42:52.031729 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.031557    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs\") pod \"dded2457-2a7c-47f5-b50a-17f672a35c8e\" (UID: \"dded2457-2a7c-47f5-b50a-17f672a35c8e\") "
Sep 15 07:42:52.031821 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.031791    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dded2457-2a7c-47f5-b50a-17f672a35c8e" (UID: "dded2457-2a7c-47f5-b50a-17f672a35c8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:42:52.032078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.032050    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dded2457-2a7c-47f5-b50a-17f672a35c8e-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.033686 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.033662    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dded2457-2a7c-47f5-b50a-17f672a35c8e" (UID: "dded2457-2a7c-47f5-b50a-17f672a35c8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:42:52.034006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.033974    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dded2457-2a7c-47f5-b50a-17f672a35c8e" (UID: "dded2457-2a7c-47f5-b50a-17f672a35c8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:42:52.034150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.034095    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf" (OuterVolumeSpecName: "kube-api-access-rnppf") pod "dded2457-2a7c-47f5-b50a-17f672a35c8e" (UID: "dded2457-2a7c-47f5-b50a-17f672a35c8e"). InnerVolumeSpecName "kube-api-access-rnppf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:42:52.069864 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.069445    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:52.069864 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.069513    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:52.132752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.132723    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.132752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.132747    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnppf\" (UniqueName: \"kubernetes.io/projected/dded2457-2a7c-47f5-b50a-17f672a35c8e-kube-api-access-rnppf\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.132752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.132757    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dded2457-2a7c-47f5-b50a-17f672a35c8e-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:42:52.616490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.616457    2574 generic.go:356] "Generic (PLEG): container finished" podID="dded2457-2a7c-47f5-b50a-17f672a35c8e" containerID="7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d" exitCode=0
Sep 15 07:42:52.616673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.616502    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"
Sep 15 07:42:52.616673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.616542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" event={"ID":"dded2457-2a7c-47f5-b50a-17f672a35c8e","Type":"ContainerDied","Data":"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"}
Sep 15 07:42:52.616673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.616593    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54" event={"ID":"dded2457-2a7c-47f5-b50a-17f672a35c8e","Type":"ContainerDied","Data":"aa5013961e9d7fdea45b7fbd58bdb756643b2f1f00627e00ec7dcc3a7045fb0c"}
Sep 15 07:42:52.616673 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.616618    2574 scope.go:117] "RemoveContainer" containerID="7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"
Sep 15 07:42:52.626293 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.626268    2574 scope.go:117] "RemoveContainer" containerID="7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"
Sep 15 07:42:52.626551 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:42:52.626533    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d\": container with ID starting with 7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d not found: ID does not exist" containerID="7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"
Sep 15 07:42:52.626600 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.626560    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d"} err="failed to get container status \"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d\": rpc error: code = NotFound desc = could not find container \"7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d\": container with ID starting with 7a847b5f6a4423de0ccf7b98f4bb60bbfd0a04e4a1c965c8f912403b6ede122d not found: ID does not exist"
Sep 15 07:42:52.632879 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.632856    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:42:52.636014 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:52.635996    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ee0096-7557794bc7-knl54"]
Sep 15 07:42:53.060962 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:53.060927    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:53.061567 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:53.061531    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:54.061638 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:54.061600    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:54.062296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:54.062263    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:54.383535 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:54.383453    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dded2457-2a7c-47f5-b50a-17f672a35c8e" path="/var/lib/kubelet/pods/dded2457-2a7c-47f5-b50a-17f672a35c8e/volumes"
Sep 15 07:42:55.061141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:55.061088    2574 patch_prober.go:28] interesting pod/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 15 07:42:55.061410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:55.061382    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 15 07:42:56.064614 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:56.064585    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:56.065003 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:56.064947    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:56.065647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:56.065633    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:42:58.493306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:58.493275    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:42:58.493715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:58.493494    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" containerID="cri-o://3f33e09acc72bffb6129cf6e42cc39aceb50b350a741452c0c3b12a3fa57a99d" gracePeriod=30
Sep 15 07:42:59.674604 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:59.674570    2574 patch_prober.go:28] interesting pod/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 07:42:59.674973 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:42:59.674630    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:43:03.693333 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.693297    2574 generic.go:356] "Generic (PLEG): container finished" podID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerID="3f33e09acc72bffb6129cf6e42cc39aceb50b350a741452c0c3b12a3fa57a99d" exitCode=0
Sep 15 07:43:03.693649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.693369    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerDied","Data":"3f33e09acc72bffb6129cf6e42cc39aceb50b350a741452c0c3b12a3fa57a99d"}
Sep 15 07:43:03.693649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.693450    2574 scope.go:117] "RemoveContainer" containerID="db1f3ef91cc70e6198d040932ae2679b041f1acff5f25d8fd79e50229ecb9f9d"
Sep 15 07:43:03.752685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.752654    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:43:03.846217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846148    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.846480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846222    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.846480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846253    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dqvq\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.846480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846281    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.846480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846306    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.846895 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846830    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.847006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846928    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:03.847006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846954    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data" (OuterVolumeSpecName: "istio-data") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:03.847499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.846894    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.847499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.847282    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:43:03.848001 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.847110    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.847222    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.848294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket\") pod \"2467ee47-5d7e-4a95-a633-e3402abe1289\" (UID: \"2467ee47-5d7e-4a95-a633-e3402abe1289\") "
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.848937    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.848960    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.848975    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.849306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.848997    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2467ee47-5d7e-4a95-a633-e3402abe1289-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.857211 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.853563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:03.857211 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.853639    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:43:03.857211 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.856444    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:03.862875 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.862838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq" (OuterVolumeSpecName: "kube-api-access-9dqvq") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "kube-api-access-9dqvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:43:03.863262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.863235    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token" (OuterVolumeSpecName: "istio-token") pod "2467ee47-5d7e-4a95-a633-e3402abe1289" (UID: "2467ee47-5d7e-4a95-a633-e3402abe1289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:43:03.949899 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.949863    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dqvq\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-kube-api-access-9dqvq\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.949899 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.949897    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.950091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.949913    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.950091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.949927    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2467ee47-5d7e-4a95-a633-e3402abe1289-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:03.950091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:03.949940    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2467ee47-5d7e-4a95-a633-e3402abe1289-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:04.700649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:04.700617    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"
Sep 15 07:43:04.701050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:04.700613    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf" event={"ID":"2467ee47-5d7e-4a95-a633-e3402abe1289","Type":"ContainerDied","Data":"b9beac6c846ea00d31e840e23931656c6d668f1b036d0c6dcdafb056bbd1693f"}
Sep 15 07:43:04.701050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:04.700755    2574 scope.go:117] "RemoveContainer" containerID="3f33e09acc72bffb6129cf6e42cc39aceb50b350a741452c0c3b12a3fa57a99d"
Sep 15 07:43:04.729269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:04.729243    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:43:04.733684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:04.733664    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ee0096-openshift-default-f8d8dbc48-bhxqf"]
Sep 15 07:43:05.809204 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:05.809138    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" podUID="beb1710f-2c29-4713-b5d0-050da3bbee64" containerName="main" containerID="cri-o://a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3" gracePeriod=45
Sep 15 07:43:06.069078 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.069017    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:43:06.174289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174258    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174438 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174344    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174438 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174394    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lrt2\" (UniqueName: \"kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174438 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174427    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174455    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174490    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs\") pod \"beb1710f-2c29-4713-b5d0-050da3bbee64\" (UID: \"beb1710f-2c29-4713-b5d0-050da3bbee64\") "
Sep 15 07:43:06.174798 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174613    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home" (OuterVolumeSpecName: "home") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:06.174798 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174681    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache" (OuterVolumeSpecName: "model-cache") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:06.174901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174830    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:06.174901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174886    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.175011 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.174904    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.176672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.176649    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2" (OuterVolumeSpecName: "kube-api-access-4lrt2") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "kube-api-access-4lrt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:43:06.176793 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.176686    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:43:06.176793 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.176726    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm" (OuterVolumeSpecName: "dshm") pod "beb1710f-2c29-4713-b5d0-050da3bbee64" (UID: "beb1710f-2c29-4713-b5d0-050da3bbee64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:43:06.223199 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.223168    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:43:06.276343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.276316    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrt2\" (UniqueName: \"kubernetes.io/projected/beb1710f-2c29-4713-b5d0-050da3bbee64-kube-api-access-4lrt2\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.276343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.276341    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.276561 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.276352    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/beb1710f-2c29-4713-b5d0-050da3bbee64-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.276561 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.276361    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/beb1710f-2c29-4713-b5d0-050da3bbee64-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:06.380407 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.380334    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" path="/var/lib/kubelet/pods/2467ee47-5d7e-4a95-a633-e3402abe1289/volumes"
Sep 15 07:43:06.713751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.713715    2574 generic.go:356] "Generic (PLEG): container finished" podID="beb1710f-2c29-4713-b5d0-050da3bbee64" containerID="a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3" exitCode=0
Sep 15 07:43:06.713751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.713753    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"
Sep 15 07:43:06.714007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.713799    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" event={"ID":"beb1710f-2c29-4713-b5d0-050da3bbee64","Type":"ContainerDied","Data":"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"}
Sep 15 07:43:06.714007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.713845    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p" event={"ID":"beb1710f-2c29-4713-b5d0-050da3bbee64","Type":"ContainerDied","Data":"aa22278d49cda1311f2f8060d00084136c3784130e1a8f3e35c9f2f63d40fe5a"}
Sep 15 07:43:06.714007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.713869    2574 scope.go:117] "RemoveContainer" containerID="a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"
Sep 15 07:43:06.722767 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.722736    2574 scope.go:117] "RemoveContainer" containerID="a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"
Sep 15 07:43:06.723022 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:43:06.723002    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3\": container with ID starting with a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3 not found: ID does not exist" containerID="a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"
Sep 15 07:43:06.723076 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.723031    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3"} err="failed to get container status \"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3\": rpc error: code = NotFound desc = could not find container \"a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3\": container with ID starting with a7fa19cf6434d32b2e36127aaed252a469edf173414c4936879f44ef88fe0ca3 not found: ID does not exist"
Sep 15 07:43:06.756995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.756967    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:43:06.762528 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:06.762505    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-c3abd1/noten-model-c3abd1-kserve-58857fbbff-ccc6p"]
Sep 15 07:43:08.379570 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:08.379537    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beb1710f-2c29-4713-b5d0-050da3bbee64" path="/var/lib/kubelet/pods/beb1710f-2c29-4713-b5d0-050da3bbee64/volumes"
Sep 15 07:43:09.296559 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.296523    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:43:09.296809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.296784    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" podUID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" containerName="maas-api" containerID="cri-o://b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3" gracePeriod=30
Sep 15 07:43:09.549512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.549175    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:43:09.603647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.603615    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs\") pod \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") "
Sep 15 07:43:09.603805 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.603655    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") pod \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") "
Sep 15 07:43:09.603805 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.603730    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca\") pod \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") "
Sep 15 07:43:09.603924 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.603889    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjgf\" (UniqueName: \"kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf\") pod \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\" (UID: \"7358ceb2-d8d3-4c11-bbe2-974b491e180d\") "
Sep 15 07:43:09.604214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.604178    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7358ceb2-d8d3-4c11-bbe2-974b491e180d" (UID: "7358ceb2-d8d3-4c11-bbe2-974b491e180d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:43:09.605856 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.605827    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7358ceb2-d8d3-4c11-bbe2-974b491e180d" (UID: "7358ceb2-d8d3-4c11-bbe2-974b491e180d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:43:09.606002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.605974    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf" (OuterVolumeSpecName: "kube-api-access-pvjgf") pod "7358ceb2-d8d3-4c11-bbe2-974b491e180d" (UID: "7358ceb2-d8d3-4c11-bbe2-974b491e180d"). InnerVolumeSpecName "kube-api-access-pvjgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:43:09.606002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.605988    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7358ceb2-d8d3-4c11-bbe2-974b491e180d" (UID: "7358ceb2-d8d3-4c11-bbe2-974b491e180d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:43:09.705693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.705659    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjgf\" (UniqueName: \"kubernetes.io/projected/7358ceb2-d8d3-4c11-bbe2-974b491e180d-kube-api-access-pvjgf\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.705693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.705692    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.705989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.705704    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7358ceb2-d8d3-4c11-bbe2-974b491e180d-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.705989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.705716    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7358ceb2-d8d3-4c11-bbe2-974b491e180d-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:43:09.733059 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.733020    2574 generic.go:356] "Generic (PLEG): container finished" podID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" containerID="b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3" exitCode=0
Sep 15 07:43:09.733239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.733067    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"
Sep 15 07:43:09.733239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.733110    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" event={"ID":"7358ceb2-d8d3-4c11-bbe2-974b491e180d","Type":"ContainerDied","Data":"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"}
Sep 15 07:43:09.733239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.733210    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8" event={"ID":"7358ceb2-d8d3-4c11-bbe2-974b491e180d","Type":"ContainerDied","Data":"1e17191464d1416a51b880d422cd4fbc810dc2c3dad8485c65778649d53057b7"}
Sep 15 07:43:09.733239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.733240    2574 scope.go:117] "RemoveContainer" containerID="b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"
Sep 15 07:43:09.743943 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.743923    2574 scope.go:117] "RemoveContainer" containerID="b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"
Sep 15 07:43:09.744227 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:43:09.744211    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3\": container with ID starting with b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3 not found: ID does not exist" containerID="b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"
Sep 15 07:43:09.744292 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.744234    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3"} err="failed to get container status \"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3\": rpc error: code = NotFound desc = could not find container \"b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3\": container with ID starting with b781804b4a914886da0013e01e107737bb655d0e855caab09450e7044505faa3 not found: ID does not exist"
Sep 15 07:43:09.758140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.757194    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:43:09.760273 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:09.760246    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fcd453-86bc6bfb86-vdzj8"]
Sep 15 07:43:10.378857 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:10.378820    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" path="/var/lib/kubelet/pods/7358ceb2-d8d3-4c11-bbe2-974b491e180d/volumes"
Sep 15 07:43:17.497375 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497341    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497730    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497740    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497751    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" containerName="maas-api"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497756    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" containerName="maas-api"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497766    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dded2457-2a7c-47f5-b50a-17f672a35c8e" containerName="maas-api"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497772    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dded2457-2a7c-47f5-b50a-17f672a35c8e" containerName="maas-api"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497783    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb1710f-2c29-4713-b5d0-050da3bbee64" containerName="main"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497789    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb1710f-2c29-4713-b5d0-050da3bbee64" containerName="main"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497796    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.497815 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497801    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.498153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497869    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dded2457-2a7c-47f5-b50a-17f672a35c8e" containerName="maas-api"
Sep 15 07:43:17.498153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497877    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7358ceb2-d8d3-4c11-bbe2-974b491e180d" containerName="maas-api"
Sep 15 07:43:17.498153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497886    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.498153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497892    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb1710f-2c29-4713-b5d0-050da3bbee64" containerName="main"
Sep 15 07:43:17.498153 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.497898    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2467ee47-5d7e-4a95-a633-e3402abe1289" containerName="istio-proxy"
Sep 15 07:43:17.502460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.502442    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.504518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.504494    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b21d1ff2\"/\"kube-root-ca.crt\""
Sep 15 07:43:17.504652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.504494    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b21d1ff2\"/\"default-dockercfg-5tb5k\""
Sep 15 07:43:17.505160 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.505136    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b21d1ff2\"/\"openshift-service-ca.crt\""
Sep 15 07:43:17.505256 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.505216    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b21d1ff2\"/\"e2e-lifecycle-model-b21d1ff2-kserve-self-signed-certs\""
Sep 15 07:43:17.509563 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.509542    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:43:17.576511 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576487    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdnlk\" (UniqueName: \"kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.576617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576521    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.576617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.576617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576591    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.576732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576679    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.576770 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.576732    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678181 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678155    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdnlk\" (UniqueName: \"kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678320    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678544 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678806 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678755    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.678928 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.678841    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.679251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.679222    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.680840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.680809    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.680840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.680827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.689449 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.689427    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdnlk\" (UniqueName: \"kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk\") pod \"e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") " pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:17.813353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:17.813252    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:18.090867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:18.090835    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:43:18.092178 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:43:18.092148    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e27451e_3068_44b9_ae2b_e17757feb004.slice/crio-7cdfe39c7d42b1d388d2d26f52469b5d1ce6fc63e9cf41a30b8a6824f1c4717d WatchSource:0}: Error finding container 7cdfe39c7d42b1d388d2d26f52469b5d1ce6fc63e9cf41a30b8a6824f1c4717d: Status 404 returned error can't find the container with id 7cdfe39c7d42b1d388d2d26f52469b5d1ce6fc63e9cf41a30b8a6824f1c4717d
Sep 15 07:43:18.797433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:18.797391    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" event={"ID":"6e27451e-3068-44b9-ae2b-e17757feb004","Type":"ContainerStarted","Data":"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"}
Sep 15 07:43:18.797801 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:18.797445    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" event={"ID":"6e27451e-3068-44b9-ae2b-e17757feb004","Type":"ContainerStarted","Data":"7cdfe39c7d42b1d388d2d26f52469b5d1ce6fc63e9cf41a30b8a6824f1c4717d"}
Sep 15 07:43:18.814845 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:18.814791    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" podStartSLOduration=1.814776162 podStartE2EDuration="1.814776162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:43:18.813485852 +0000 UTC m=+1486.907652119" watchObservedRunningTime="2026-09-15 07:43:18.814776162 +0000 UTC m=+1486.908942424"
Sep 15 07:43:24.613781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.610755    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:43:24.616371 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.616344    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.621195 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.621169    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-04464b-serving-cert\""
Sep 15 07:43:24.624147 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.622546    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:43:24.747178 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.747144    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.747320 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.747248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.747370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.747335    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.747412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.747389    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52zzx\" (UniqueName: \"kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.848173 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.848137    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.848335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.848209    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52zzx\" (UniqueName: \"kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.848335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.848307    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.848476 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.848398    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.849091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.849060    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.851209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.851183    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.851348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.851318    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.857509 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.857487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52zzx\" (UniqueName: \"kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx\") pod \"maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:24.933169 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:24.933147    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:25.206084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:25.206055    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:43:25.207359 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:43:25.207331    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f065738_983d_49d6_98fb_0906ed0492cd.slice/crio-9c4e76de402e3c41be2c1d4ee136809289f13a5bd97c70633dd370d9d0f71533 WatchSource:0}: Error finding container 9c4e76de402e3c41be2c1d4ee136809289f13a5bd97c70633dd370d9d0f71533: Status 404 returned error can't find the container with id 9c4e76de402e3c41be2c1d4ee136809289f13a5bd97c70633dd370d9d0f71533
Sep 15 07:43:25.841643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:25.841539    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" event={"ID":"2f065738-983d-49d6-98fb-0906ed0492cd","Type":"ContainerStarted","Data":"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"}
Sep 15 07:43:25.842044 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:25.841709    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:25.842044 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:25.841768    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" event={"ID":"2f065738-983d-49d6-98fb-0906ed0492cd","Type":"ContainerStarted","Data":"9c4e76de402e3c41be2c1d4ee136809289f13a5bd97c70633dd370d9d0f71533"}
Sep 15 07:43:25.859274 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:25.859233    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" podStartSLOduration=1.478457648 podStartE2EDuration="1.8592185s" totalImagesPullingTime="380.760852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:43:25.857762655 +0000 UTC m=+1493.951928918" watchObservedRunningTime="2026-09-15 07:43:25.8592185 +0000 UTC m=+1493.953384772"
Sep 15 07:43:27.814299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:27.814255    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:27.814695 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:27.814309    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:27.826807 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:27.826782    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:27.865239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:27.865214    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:43:31.854529 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:31.854497    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:43:32.557277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:32.557242    2574 scope.go:117] "RemoveContainer" containerID="5d9aa50df8476bc2d1242e03a71b5137def27de125e00e850b0a17b4ddece136"
Sep 15 07:43:43.016820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:43.016780    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:43:59.026956 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.026879    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:43:59.031903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.031875    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.034269 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.034243    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e3a482-serving-cert\""
Sep 15 07:43:59.044029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.044005    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:43:59.150031 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.149993    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.150227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.150045    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.150227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.150150    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.150227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.150214    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcdqw\" (UniqueName: \"kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.251501 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.251468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.251640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.251510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.251640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.251546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.251640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.251595    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcdqw\" (UniqueName: \"kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.251795 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:43:59.251704    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-serving-cert: secret "maas-api-e2e-shared-b-w3-e3a482-serving-cert" not found
Sep 15 07:43:59.251845 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:43:59.251795    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls podName:0201f423-a2d8-4fed-b139-baa22b77424e nodeName:}" failed. No retries permitted until 2026-09-15 07:43:59.751772416 +0000 UTC m=+1527.845938656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" (UID: "0201f423-a2d8-4fed-b139-baa22b77424e") : secret "maas-api-e2e-shared-b-w3-e3a482-serving-cert" not found
Sep 15 07:43:59.252397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.252355    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.254284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.254262    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.262008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.261985    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcdqw\" (UniqueName: \"kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.757325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.757280    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.760698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.760668    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:43:59.945490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:43:59.945453    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:44:00.219997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:00.219964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:44:00.221095 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:44:00.221071    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0201f423_a2d8_4fed_b139_baa22b77424e.slice/crio-53ded2f18a5386b22b4708dae0a9ba0ce88b520ac30c7725f62beb617776e15a WatchSource:0}: Error finding container 53ded2f18a5386b22b4708dae0a9ba0ce88b520ac30c7725f62beb617776e15a: Status 404 returned error can't find the container with id 53ded2f18a5386b22b4708dae0a9ba0ce88b520ac30c7725f62beb617776e15a
Sep 15 07:44:01.064052 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:01.064015    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" event={"ID":"0201f423-a2d8-4fed-b139-baa22b77424e","Type":"ContainerStarted","Data":"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"}
Sep 15 07:44:01.064258 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:01.064089    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" event={"ID":"0201f423-a2d8-4fed-b139-baa22b77424e","Type":"ContainerStarted","Data":"53ded2f18a5386b22b4708dae0a9ba0ce88b520ac30c7725f62beb617776e15a"}
Sep 15 07:44:01.064258 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:01.064248    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:44:01.081248 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:01.081194    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" podStartSLOduration=1.688625762 podStartE2EDuration="2.081181286s" totalImagesPullingTime="392.555524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:44:01.07895553 +0000 UTC m=+1529.173121793" watchObservedRunningTime="2026-09-15 07:44:01.081181286 +0000 UTC m=+1529.175347547"
Sep 15 07:44:07.075828 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:07.075798    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:44:09.218920 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:09.218885    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:44:50.317574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:44:50.317539    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:45:25.015711 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:45:25.015614    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:45:50.719757 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:45:50.719707    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:46:14.791822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:14.791779    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:46:14.792566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:14.792531    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" podUID="d00749b8-f119-47d3-beee-85091b312d52" containerName="maas-api" containerID="cri-o://ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f" gracePeriod=30
Sep 15 07:46:15.081498 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.081459    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:46:15.245772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.245727    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs\") pod \"d00749b8-f119-47d3-beee-85091b312d52\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") "
Sep 15 07:46:15.245974 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.245846    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8zcj\" (UniqueName: \"kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj\") pod \"d00749b8-f119-47d3-beee-85091b312d52\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") "
Sep 15 07:46:15.245974 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.245903    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") pod \"d00749b8-f119-47d3-beee-85091b312d52\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") "
Sep 15 07:46:15.246104 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.245998    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca\") pod \"d00749b8-f119-47d3-beee-85091b312d52\" (UID: \"d00749b8-f119-47d3-beee-85091b312d52\") "
Sep 15 07:46:15.246590 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.246559    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d00749b8-f119-47d3-beee-85091b312d52" (UID: "d00749b8-f119-47d3-beee-85091b312d52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:46:15.258334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.250283    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d00749b8-f119-47d3-beee-85091b312d52" (UID: "d00749b8-f119-47d3-beee-85091b312d52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:46:15.258334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.254594    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj" (OuterVolumeSpecName: "kube-api-access-h8zcj") pod "d00749b8-f119-47d3-beee-85091b312d52" (UID: "d00749b8-f119-47d3-beee-85091b312d52"). InnerVolumeSpecName "kube-api-access-h8zcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:46:15.263508 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.263201    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d00749b8-f119-47d3-beee-85091b312d52" (UID: "d00749b8-f119-47d3-beee-85091b312d52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:46:15.347809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.347643    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:15.347809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.347687    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zcj\" (UniqueName: \"kubernetes.io/projected/d00749b8-f119-47d3-beee-85091b312d52-kube-api-access-h8zcj\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:15.347809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.347702    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d00749b8-f119-47d3-beee-85091b312d52-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:15.347809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.347716    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d00749b8-f119-47d3-beee-85091b312d52-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:15.897817 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.897778    2574 generic.go:356] "Generic (PLEG): container finished" podID="d00749b8-f119-47d3-beee-85091b312d52" containerID="ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f" exitCode=0
Sep 15 07:46:15.898344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.897829    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"
Sep 15 07:46:15.898344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.897873    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" event={"ID":"d00749b8-f119-47d3-beee-85091b312d52","Type":"ContainerDied","Data":"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"}
Sep 15 07:46:15.898344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.897939    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2" event={"ID":"d00749b8-f119-47d3-beee-85091b312d52","Type":"ContainerDied","Data":"7886959abe35d7aa7715b995f699a7842769791496727efcf162f2af71c54251"}
Sep 15 07:46:15.898344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.897975    2574 scope.go:117] "RemoveContainer" containerID="ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"
Sep 15 07:46:15.908534 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.908500    2574 scope.go:117] "RemoveContainer" containerID="ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"
Sep 15 07:46:15.908938 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:46:15.908911    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f\": container with ID starting with ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f not found: ID does not exist" containerID="ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"
Sep 15 07:46:15.909037 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.908946    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f"} err="failed to get container status \"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f\": rpc error: code = NotFound desc = could not find container \"ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f\": container with ID starting with ee2cf3e94f5f54fa5d6ffd820f2a769e27b87d4d8d848d716e86a83052d4363f not found: ID does not exist"
Sep 15 07:46:15.919743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.919703    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:46:15.925585 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:15.925551    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b21d1ff2-8646b46c8d-c22f2"]
Sep 15 07:46:16.381149 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:16.381087    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d00749b8-f119-47d3-beee-85091b312d52" path="/var/lib/kubelet/pods/d00749b8-f119-47d3-beee-85091b312d52/volumes"
Sep 15 07:46:23.059937 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:23.059895    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:46:35.507729 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:35.507685    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:46:38.080197 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.080157    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" podUID="6e27451e-3068-44b9-ae2b-e17757feb004" containerName="main" containerID="cri-o://04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3" gracePeriod=45
Sep 15 07:46:38.355418 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.355382    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:46:38.491058 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491015    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdnlk\" (UniqueName: \"kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491155    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491188    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491211    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491296 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs\") pod \"6e27451e-3068-44b9-ae2b-e17757feb004\" (UID: \"6e27451e-3068-44b9-ae2b-e17757feb004\") "
Sep 15 07:46:38.491575 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:38.491575 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491442    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:38.491575 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491541    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home" (OuterVolumeSpecName: "home") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:38.491684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491648    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:38.491684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491668    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:38.491684 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.491681    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:38.493572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.493533    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm" (OuterVolumeSpecName: "dshm") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:46:38.493572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.493563    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:46:38.493744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.493606    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk" (OuterVolumeSpecName: "kube-api-access-wdnlk") pod "6e27451e-3068-44b9-ae2b-e17757feb004" (UID: "6e27451e-3068-44b9-ae2b-e17757feb004"). InnerVolumeSpecName "kube-api-access-wdnlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:46:38.593171 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.593029    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdnlk\" (UniqueName: \"kubernetes.io/projected/6e27451e-3068-44b9-ae2b-e17757feb004-kube-api-access-wdnlk\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:38.593171 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.593067    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27451e-3068-44b9-ae2b-e17757feb004-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:38.593171 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:38.593079    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27451e-3068-44b9-ae2b-e17757feb004-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:46:39.039452 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.039416    2574 generic.go:356] "Generic (PLEG): container finished" podID="6e27451e-3068-44b9-ae2b-e17757feb004" containerID="04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3" exitCode=0
Sep 15 07:46:39.039672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.039459    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"
Sep 15 07:46:39.039672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.039506    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" event={"ID":"6e27451e-3068-44b9-ae2b-e17757feb004","Type":"ContainerDied","Data":"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"}
Sep 15 07:46:39.039672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.039559    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh" event={"ID":"6e27451e-3068-44b9-ae2b-e17757feb004","Type":"ContainerDied","Data":"7cdfe39c7d42b1d388d2d26f52469b5d1ce6fc63e9cf41a30b8a6824f1c4717d"}
Sep 15 07:46:39.039672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.039583    2574 scope.go:117] "RemoveContainer" containerID="04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"
Sep 15 07:46:39.050088 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.050059    2574 scope.go:117] "RemoveContainer" containerID="04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"
Sep 15 07:46:39.050571 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:46:39.050544    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3\": container with ID starting with 04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3 not found: ID does not exist" containerID="04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"
Sep 15 07:46:39.050683 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.050589    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3"} err="failed to get container status \"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3\": rpc error: code = NotFound desc = could not find container \"04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3\": container with ID starting with 04ed532f7ef59b520268724ca9f86758f61973b1404a37e857c71f61a42652e3 not found: ID does not exist"
Sep 15 07:46:39.062322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.062280    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:46:39.066250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:39.066215    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b21d1ff2/e2e-lifecycle-model-b21d1ff2-kserve-78596f6c55-x5bbh"]
Sep 15 07:46:40.379530 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:40.379490    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e27451e-3068-44b9-ae2b-e17757feb004" path="/var/lib/kubelet/pods/6e27451e-3068-44b9-ae2b-e17757feb004/volumes"
Sep 15 07:46:42.511409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.511363    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:46:42.511997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.511975    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d00749b8-f119-47d3-beee-85091b312d52" containerName="maas-api"
Sep 15 07:46:42.511997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.511996    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d00749b8-f119-47d3-beee-85091b312d52" containerName="maas-api"
Sep 15 07:46:42.512209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.512012    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e27451e-3068-44b9-ae2b-e17757feb004" containerName="main"
Sep 15 07:46:42.512209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.512018    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e27451e-3068-44b9-ae2b-e17757feb004" containerName="main"
Sep 15 07:46:42.512209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.512108    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e27451e-3068-44b9-ae2b-e17757feb004" containerName="main"
Sep 15 07:46:42.512209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.512141    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d00749b8-f119-47d3-beee-85091b312d52" containerName="maas-api"
Sep 15 07:46:42.516742 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.516698    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.519446 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.519418    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-2735c5\"/\"embedding-w0-2735c5-kserve-self-signed-certs\""
Sep 15 07:46:42.522483 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.522455    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:46:42.634706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.634660    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.634706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.634706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phz4q\" (UniqueName: \"kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.634997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.634833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.634997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.634922    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.634997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.634957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.635110 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.635028    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736352    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736399 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736401    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736457    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736482    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736501    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phz4q\" (UniqueName: \"kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736643 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736575    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.736948 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736918    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.737063 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736957    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.737063 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.736969    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.738899 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.738876    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.739041 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.739025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.745940 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.745913    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phz4q\" (UniqueName: \"kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q\") pod \"embedding-w0-2735c5-kserve-67599654b5-2wxwh\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") " pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:42.828253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:42.828108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:43.106306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:43.106270    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:46:44.073639 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:44.073596    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" event={"ID":"692b5255-6265-4b38-8585-c2bf17ebf066","Type":"ContainerStarted","Data":"e0b036ba659e8015a0fa9d0e9807f1699700d0df5f821cdce827827f814a489e"}
Sep 15 07:46:44.074147 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:44.073651    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" event={"ID":"692b5255-6265-4b38-8585-c2bf17ebf066","Type":"ContainerStarted","Data":"81d1966a3075e9cc2aed923406dae866dbb8bb48b12311789f4303d460d05114"}
Sep 15 07:46:44.090179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:44.090088    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" podStartSLOduration=2.090070173 podStartE2EDuration="2.090070173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:46:44.089642829 +0000 UTC m=+1692.183809091" watchObservedRunningTime="2026-09-15 07:46:44.090070173 +0000 UTC m=+1692.184236445"
Sep 15 07:46:52.828298 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:52.828257    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:52.828898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:52.828324    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:52.841369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:52.841337    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:46:53.138919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:46:53.138832    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:47:11.769401 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:11.769300    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:47:11.769867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:11.769549    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy" containerID="cri-o://57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab" gracePeriod=30
Sep 15 07:47:13.136141 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:13.136075    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:47:17.041083 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.041016    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:47:17.118484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118442    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118496    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118532    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118583    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118613    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldrqr\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118652    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118720    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.118945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118833    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:47:17.118945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118853    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data" (OuterVolumeSpecName: "istio-data") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:47:17.119109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.118961    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:47:17.119213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.119192    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.119263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.119219    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.119263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.119239    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.121054 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.121008    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:47:17.121223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.121132    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr" (OuterVolumeSpecName: "kube-api-access-ldrqr") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "kube-api-access-ldrqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:47:17.121383 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.121349    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token" (OuterVolumeSpecName: "istio-token") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:47:17.121383 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.121367    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:47:17.219555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219525    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.219555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219560    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs\") pod \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\" (UID: \"44dab330-9a2f-48e0-ae9f-6abe663b5a69\") "
Sep 15 07:47:17.219820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219797    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.219820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219808    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrqr\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-kube-api-access-ldrqr\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.219820 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219818    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.219942 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219826    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44dab330-9a2f-48e0-ae9f-6abe663b5a69-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.219942 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219817    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:47:17.219942 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.219844    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "44dab330-9a2f-48e0-ae9f-6abe663b5a69" (UID: "44dab330-9a2f-48e0-ae9f-6abe663b5a69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:47:17.282428 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.282389    2574 generic.go:356] "Generic (PLEG): container finished" podID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerID="57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab" exitCode=0
Sep 15 07:47:17.282627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.282434    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"
Sep 15 07:47:17.282627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.282473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerDied","Data":"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"}
Sep 15 07:47:17.282627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.282542    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6" event={"ID":"44dab330-9a2f-48e0-ae9f-6abe663b5a69","Type":"ContainerDied","Data":"9645311b6b4b4d6a8f861cb01198781ad9eed37008ebda650bc350e180648e72"}
Sep 15 07:47:17.282627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.282575    2574 scope.go:117] "RemoveContainer" containerID="57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"
Sep 15 07:47:17.293249 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.293217    2574 scope.go:117] "RemoveContainer" containerID="5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"
Sep 15 07:47:17.303199 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.303178    2574 scope.go:117] "RemoveContainer" containerID="57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"
Sep 15 07:47:17.303606 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:47:17.303584    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab\": container with ID starting with 57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab not found: ID does not exist" containerID="57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"
Sep 15 07:47:17.303674 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.303616    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab"} err="failed to get container status \"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab\": rpc error: code = NotFound desc = could not find container \"57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab\": container with ID starting with 57af0afdbe69aee18068eaef6674afac262c634bc5a4db69da59bb57810038ab not found: ID does not exist"
Sep 15 07:47:17.303674 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.303636    2574 scope.go:117] "RemoveContainer" containerID="5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"
Sep 15 07:47:17.303941 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:47:17.303921    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0\": container with ID starting with 5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0 not found: ID does not exist" containerID="5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"
Sep 15 07:47:17.304000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.303949    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:47:17.304000 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.303952    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0"} err="failed to get container status \"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0\": rpc error: code = NotFound desc = could not find container \"5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0\": container with ID starting with 5cba037573b1df5a0f0b72efd177fe21c8291c85994a21e2ca0940def4841ca0 not found: ID does not exist"
Sep 15 07:47:17.308292 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.308251    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b21d1ff2-openshift-default-5d6cc9f98d-pqsl6"]
Sep 15 07:47:17.320822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.320786    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:17.320822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:17.320816    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44dab330-9a2f-48e0-ae9f-6abe663b5a69-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:47:18.380343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:18.380305    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" path="/var/lib/kubelet/pods/44dab330-9a2f-48e0-ae9f-6abe663b5a69/volumes"
Sep 15 07:47:48.106108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:47:48.106067    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:48:06.017868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.017828    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:48:06.018552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.018528    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:06.018598 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.018558    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:06.018707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.018694    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:06.018743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.018713    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:06.022338 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.022307    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.024472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.024448    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4ef86b-openshift-default-dockercfg-fn9mj\""
Sep 15 07:48:06.032084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.032052    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:48:06.123641 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123600    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.123830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123654    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.123830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123701    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.123830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.123830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123773    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.123830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123814    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.124135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.124135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123857    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjdj\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.124135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.123893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224584    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224633 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224632    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjdj\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224692    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224724    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224757    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.224919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224936    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.224997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225326 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.225235    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225326 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.225306    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.225352    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225406 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.225380    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.225565 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.225539    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.227343 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.227320    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.227491 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.227469    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.232533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.232494    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.232685 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.232626    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjdj\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj\") pod \"e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") " pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.334939 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.334828    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:06.621523 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.621489    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:48:06.623436 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:48:06.623401    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd91a6cd_a760_4ed7_bdc0_b475c791c621.slice/crio-d9e11957d443f04d41718e7a529cf58855a8e5c8a92ba71671a2bac727a61904 WatchSource:0}: Error finding container d9e11957d443f04d41718e7a529cf58855a8e5c8a92ba71671a2bac727a61904: Status 404 returned error can't find the container with id d9e11957d443f04d41718e7a529cf58855a8e5c8a92ba71671a2bac727a61904
Sep 15 07:48:06.626152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.626096    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:06.626234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.626197    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:06.626234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:06.626225    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:07.594519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:07.594476    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerStarted","Data":"a3076cab4494d8d0a5e00a0e9dfe48250007fdfffe5d02e4b4a9980ee7d8d57b"}
Sep 15 07:48:07.594967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:07.594534    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerStarted","Data":"d9e11957d443f04d41718e7a529cf58855a8e5c8a92ba71671a2bac727a61904"}
Sep 15 07:48:07.615359 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:07.615297    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podStartSLOduration=2.615279385 podStartE2EDuration="2.615279385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:07.611871122 +0000 UTC m=+1775.706037384" watchObservedRunningTime="2026-09-15 07:48:07.615279385 +0000 UTC m=+1775.709445638"
Sep 15 07:48:08.335460 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:08.335415    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:08.341417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:08.341384    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:08.601611 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:08.601524    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:08.602764 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:08.602739    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:14.501682 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.501627    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:48:14.516471 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.516433    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:48:14.532336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.532298    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:48:14.713281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.713245    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:48:14.713715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.713701    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:14.713762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.713717    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="44dab330-9a2f-48e0-ae9f-6abe663b5a69" containerName="istio-proxy"
Sep 15 07:48:14.717328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.717302    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.719583 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.719556    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-4ef86b-serving-cert\""
Sep 15 07:48:14.731151 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.731034    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:48:14.820179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.820065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.820179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.820142    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.820447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.820227    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.820447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.820276    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nc5c\" (UniqueName: \"kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.921456 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.921407    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.921669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.921470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.921669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.921536    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.921669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.921581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nc5c\" (UniqueName: \"kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.922238 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.922210    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.923950 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.923919    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.924050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.923958    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:14.933480 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:14.933453    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nc5c\" (UniqueName: \"kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c\") pod \"maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:15.033777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:15.033742    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:15.329881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:15.329852    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:48:15.332357 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:48:15.332328    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c8beb6_113e_4c39_8252_85b944ded485.slice/crio-a484d250c9fd232efaf78835b047816a9cc937bdff8763ba745d0b5f1ec8688f WatchSource:0}: Error finding container a484d250c9fd232efaf78835b047816a9cc937bdff8763ba745d0b5f1ec8688f: Status 404 returned error can't find the container with id a484d250c9fd232efaf78835b047816a9cc937bdff8763ba745d0b5f1ec8688f
Sep 15 07:48:15.333824 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:15.333800    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:48:15.655069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:15.655031    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" event={"ID":"22c8beb6-113e-4c39-8252-85b944ded485","Type":"ContainerStarted","Data":"a484d250c9fd232efaf78835b047816a9cc937bdff8763ba745d0b5f1ec8688f"}
Sep 15 07:48:17.915932 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:17.915887    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:48:18.675361 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:18.675324    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" event={"ID":"22c8beb6-113e-4c39-8252-85b944ded485","Type":"ContainerStarted","Data":"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"}
Sep 15 07:48:18.675586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:18.675412    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:18.695039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:18.694980    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" podStartSLOduration=1.958820116 podStartE2EDuration="4.694962488s" totalImagesPullingTime="2.736142372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:18.693205288 +0000 UTC m=+1786.787371547" watchObservedRunningTime="2026-09-15 07:48:18.694962488 +0000 UTC m=+1786.789128750"
Sep 15 07:48:24.688441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:24.688400    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:48:25.674481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:25.674442    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:48:25.674725 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:25.674698    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" podUID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" containerName="maas-api" containerID="cri-o://4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854" gracePeriod=30
Sep 15 07:48:25.943714 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:25.943684    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:48:26.039168 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.039128    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44ds4\" (UniqueName: \"kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4\") pod \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") "
Sep 15 07:48:26.039334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.039254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") pod \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") "
Sep 15 07:48:26.039334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.039291    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs\") pod \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") "
Sep 15 07:48:26.039334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.039327    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca\") pod \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\" (UID: \"c42acb8e-dc2b-46f9-9d01-be252b9209b8\") "
Sep 15 07:48:26.039710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.039682    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c42acb8e-dc2b-46f9-9d01-be252b9209b8" (UID: "c42acb8e-dc2b-46f9-9d01-be252b9209b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:48:26.041447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.041419    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c42acb8e-dc2b-46f9-9d01-be252b9209b8" (UID: "c42acb8e-dc2b-46f9-9d01-be252b9209b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:26.041552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.041437    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c42acb8e-dc2b-46f9-9d01-be252b9209b8" (UID: "c42acb8e-dc2b-46f9-9d01-be252b9209b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:26.041552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.041447    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4" (OuterVolumeSpecName: "kube-api-access-44ds4") pod "c42acb8e-dc2b-46f9-9d01-be252b9209b8" (UID: "c42acb8e-dc2b-46f9-9d01-be252b9209b8"). InnerVolumeSpecName "kube-api-access-44ds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:26.140522 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.140479    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.140522 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.140516    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c42acb8e-dc2b-46f9-9d01-be252b9209b8-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.140522 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.140525    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42acb8e-dc2b-46f9-9d01-be252b9209b8-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.140776 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.140536    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44ds4\" (UniqueName: \"kubernetes.io/projected/c42acb8e-dc2b-46f9-9d01-be252b9209b8-kube-api-access-44ds4\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:26.729767 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.729732    2574 generic.go:356] "Generic (PLEG): container finished" podID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" containerID="4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854" exitCode=0
Sep 15 07:48:26.729952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.729780    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"
Sep 15 07:48:26.729952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.729816    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" event={"ID":"c42acb8e-dc2b-46f9-9d01-be252b9209b8","Type":"ContainerDied","Data":"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"}
Sep 15 07:48:26.729952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.729882    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck" event={"ID":"c42acb8e-dc2b-46f9-9d01-be252b9209b8","Type":"ContainerDied","Data":"612949a56ad8f87740b5f1068b6fab386b7c751196247de3a3b53b2eb3c980de"}
Sep 15 07:48:26.729952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.729912    2574 scope.go:117] "RemoveContainer" containerID="4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"
Sep 15 07:48:26.740043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.740019    2574 scope.go:117] "RemoveContainer" containerID="4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"
Sep 15 07:48:26.740419 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:48:26.740394    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854\": container with ID starting with 4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854 not found: ID does not exist" containerID="4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"
Sep 15 07:48:26.740474 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.740429    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854"} err="failed to get container status \"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854\": rpc error: code = NotFound desc = could not find container \"4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854\": container with ID starting with 4eccc09ee5fd28e1c22eaef5362bbd31428ae3df4a87b88ee7dd954924e05854 not found: ID does not exist"
Sep 15 07:48:26.747090 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.747060    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:48:26.754344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:26.754306    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-2735c5-5f5dd4484-j4cck"]
Sep 15 07:48:28.380307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:28.380268    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" path="/var/lib/kubelet/pods/c42acb8e-dc2b-46f9-9d01-be252b9209b8/volumes"
Sep 15 07:48:29.521700 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.521653    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" podUID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" containerName="main" containerID="cri-o://7f529d43622faadccb1e9e633ae9856f9f910214648b609e430ddf43d9a4a919" gracePeriod=45
Sep 15 07:48:29.536918 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.536885    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" podUID="692b5255-6265-4b38-8585-c2bf17ebf066" containerName="main" containerID="cri-o://e0b036ba659e8015a0fa9d0e9807f1699700d0df5f821cdce827827f814a489e" gracePeriod=45
Sep 15 07:48:29.555614 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.555564    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" podUID="2a7169ef-806d-4ee7-b51c-6145af872f5f" containerName="main" containerID="cri-o://fc8c98e66ffc204697fa322719ddc36e4eb7f75f6b376b214bc1e913c5ac70b3" gracePeriod=45
Sep 15 07:48:29.755238 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.755200    2574 generic.go:356] "Generic (PLEG): container finished" podID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" containerID="7f529d43622faadccb1e9e633ae9856f9f910214648b609e430ddf43d9a4a919" exitCode=0
Sep 15 07:48:29.755419 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.755378    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" event={"ID":"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b","Type":"ContainerDied","Data":"7f529d43622faadccb1e9e633ae9856f9f910214648b609e430ddf43d9a4a919"}
Sep 15 07:48:29.757384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.757357    2574 generic.go:356] "Generic (PLEG): container finished" podID="2a7169ef-806d-4ee7-b51c-6145af872f5f" containerID="fc8c98e66ffc204697fa322719ddc36e4eb7f75f6b376b214bc1e913c5ac70b3" exitCode=0
Sep 15 07:48:29.757516 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.757439    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" event={"ID":"2a7169ef-806d-4ee7-b51c-6145af872f5f","Type":"ContainerDied","Data":"fc8c98e66ffc204697fa322719ddc36e4eb7f75f6b376b214bc1e913c5ac70b3"}
Sep 15 07:48:29.759258 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.759229    2574 generic.go:356] "Generic (PLEG): container finished" podID="692b5255-6265-4b38-8585-c2bf17ebf066" containerID="e0b036ba659e8015a0fa9d0e9807f1699700d0df5f821cdce827827f814a489e" exitCode=0
Sep 15 07:48:29.759379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.759292    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" event={"ID":"692b5255-6265-4b38-8585-c2bf17ebf066","Type":"ContainerDied","Data":"e0b036ba659e8015a0fa9d0e9807f1699700d0df5f821cdce827827f814a489e"}
Sep 15 07:48:29.884256 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.884225    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:48:29.889094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.889008    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:48:29.893389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.893360    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:48:29.981658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981625    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bgjx\" (UniqueName: \"kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.981819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981692    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.981819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981710    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.981819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981736    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.981819 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981769    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981819    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981866    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981929    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981951    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhh57\" (UniqueName: \"kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981972    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phz4q\" (UniqueName: \"kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.981997    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982024    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982065    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982091    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982148    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home\") pod \"2a7169ef-806d-4ee7-b51c-6145af872f5f\" (UID: \"2a7169ef-806d-4ee7-b51c-6145af872f5f\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982185    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache\") pod \"692b5255-6265-4b38-8585-c2bf17ebf066\" (UID: \"692b5255-6265-4b38-8585-c2bf17ebf066\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982209    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982249    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir\") pod \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\" (UID: \"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b\") "
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982287    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.982454 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982359    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home" (OuterVolumeSpecName: "home") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.982855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982552    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.982855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982562    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.982855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982570    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.982855 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.982786    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache" (OuterVolumeSpecName: "model-cache") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.983180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983157    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983393 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983377    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983508    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983651 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983638    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983783 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983771    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983875    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:29.983977 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983180    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home" (OuterVolumeSpecName: "home") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.984073 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983340    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.984184 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.983566    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home" (OuterVolumeSpecName: "home") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.985740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.985713    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm" (OuterVolumeSpecName: "dshm") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.986018 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.985935    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm" (OuterVolumeSpecName: "dshm") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:29.986018 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.985975    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57" (OuterVolumeSpecName: "kube-api-access-hhh57") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "kube-api-access-hhh57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:29.986158 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.986030    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx" (OuterVolumeSpecName: "kube-api-access-7bgjx") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "kube-api-access-7bgjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:29.986247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.986151    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:29.986345 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.986321    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" (UID: "0b8b0a55-5c0a-47b1-a38c-76a414c1c05b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:29.986345 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.986327    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q" (OuterVolumeSpecName: "kube-api-access-phz4q") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "kube-api-access-phz4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:48:29.987215 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.987191    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "692b5255-6265-4b38-8585-c2bf17ebf066" (UID: "692b5255-6265-4b38-8585-c2bf17ebf066"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:48:29.987295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:29.987261    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm" (OuterVolumeSpecName: "dshm") pod "2a7169ef-806d-4ee7-b51c-6145af872f5f" (UID: "2a7169ef-806d-4ee7-b51c-6145af872f5f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085385    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085420    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhh57\" (UniqueName: \"kubernetes.io/projected/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-kube-api-access-hhh57\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085431    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phz4q\" (UniqueName: \"kubernetes.io/projected/692b5255-6265-4b38-8585-c2bf17ebf066-kube-api-access-phz4q\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085440    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085449    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085457    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/692b5255-6265-4b38-8585-c2bf17ebf066-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085464    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085471    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7169ef-806d-4ee7-b51c-6145af872f5f-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085478    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085486    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgjx\" (UniqueName: \"kubernetes.io/projected/2a7169ef-806d-4ee7-b51c-6145af872f5f-kube-api-access-7bgjx\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085484 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085494    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7169ef-806d-4ee7-b51c-6145af872f5f-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.085950 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.085501    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/692b5255-6265-4b38-8585-c2bf17ebf066-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:48:30.768152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.768105    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"
Sep 15 07:48:30.768152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.768140    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp" event={"ID":"0b8b0a55-5c0a-47b1-a38c-76a414c1c05b","Type":"ContainerDied","Data":"d8d2e4c05c9506b200a292cb421372b5447b6b910886d292bec25608f409a0a9"}
Sep 15 07:48:30.768668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.768199    2574 scope.go:117] "RemoveContainer" containerID="7f529d43622faadccb1e9e633ae9856f9f910214648b609e430ddf43d9a4a919"
Sep 15 07:48:30.769997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.769968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp" event={"ID":"2a7169ef-806d-4ee7-b51c-6145af872f5f","Type":"ContainerDied","Data":"8defe21b35240fb3b3cbe5e47220e386225b1a854ca5b77a2aa825fd308c0767"}
Sep 15 07:48:30.770103 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.769993    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"
Sep 15 07:48:30.771676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.771656    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"
Sep 15 07:48:30.771774 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.771674    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh" event={"ID":"692b5255-6265-4b38-8585-c2bf17ebf066","Type":"ContainerDied","Data":"81d1966a3075e9cc2aed923406dae866dbb8bb48b12311789f4303d460d05114"}
Sep 15 07:48:30.783442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.783421    2574 scope.go:117] "RemoveContainer" containerID="fc8c98e66ffc204697fa322719ddc36e4eb7f75f6b376b214bc1e913c5ac70b3"
Sep 15 07:48:30.793325 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.793303    2574 scope.go:117] "RemoveContainer" containerID="e0b036ba659e8015a0fa9d0e9807f1699700d0df5f821cdce827827f814a489e"
Sep 15 07:48:30.805896 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.805865    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:48:30.825397 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.825363    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/embedding-w0-2735c5-kserve-67599654b5-2wxwh"]
Sep 15 07:48:30.852138 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.852085    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:48:30.863499 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.863464    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/premium-w0-2735c5-kserve-7fb7779f89-tlwfp"]
Sep 15 07:48:30.876273 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.876233    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:48:30.879264 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:30.879232    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-2735c5/distinct-w0-2735c5-kserve-64677d94f6-hsjbp"]
Sep 15 07:48:32.380159 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:32.380106    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" path="/var/lib/kubelet/pods/0b8b0a55-5c0a-47b1-a38c-76a414c1c05b/volumes"
Sep 15 07:48:32.380572 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:32.380558    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7169ef-806d-4ee7-b51c-6145af872f5f" path="/var/lib/kubelet/pods/2a7169ef-806d-4ee7-b51c-6145af872f5f/volumes"
Sep 15 07:48:32.380953 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:32.380941    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692b5255-6265-4b38-8585-c2bf17ebf066" path="/var/lib/kubelet/pods/692b5255-6265-4b38-8585-c2bf17ebf066/volumes"
Sep 15 07:48:35.310383 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310341    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:48:35.310944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310922    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" containerName="maas-api"
Sep 15 07:48:35.310944 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310945    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" containerName="maas-api"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310956    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692b5255-6265-4b38-8585-c2bf17ebf066" containerName="main"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310962    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="692b5255-6265-4b38-8585-c2bf17ebf066" containerName="main"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310985    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" containerName="main"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.310991    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" containerName="main"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311007    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7169ef-806d-4ee7-b51c-6145af872f5f" containerName="main"
Sep 15 07:48:35.311068 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311013    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7169ef-806d-4ee7-b51c-6145af872f5f" containerName="main"
Sep 15 07:48:35.311299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311073    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42acb8e-dc2b-46f9-9d01-be252b9209b8" containerName="maas-api"
Sep 15 07:48:35.311299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311082    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="692b5255-6265-4b38-8585-c2bf17ebf066" containerName="main"
Sep 15 07:48:35.311299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311089    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7169ef-806d-4ee7-b51c-6145af872f5f" containerName="main"
Sep 15 07:48:35.311299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.311096    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b8b0a55-5c0a-47b1-a38c-76a414c1c05b" containerName="main"
Sep 15 07:48:35.314226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.314204    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.316593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.316568    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1755f7-openshift-default-dockercfg-xv9qm\""
Sep 15 07:48:35.327216 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.327189    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:48:35.440916 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.440870    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441125 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.440981    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7d2\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441125 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441012    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441125 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441078    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441180    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441219    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441248    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.441410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.441370    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542208 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542170    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542243    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542288    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7d2\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542313    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542373    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542408    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542429    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542578    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542740    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542832    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542983 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542928    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.542983 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542973    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.543062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.542998    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.544761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.544742    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.545060 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.545041    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.550666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.550635    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.551234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.551216    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7d2\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2\") pod \"e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") " pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.625709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.625616    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:35.932188 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.932155    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:48:35.934001 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:48:35.933971    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234a3afc_9483_4f7b_a61f_3d3762322fd6.slice/crio-5754dd89d8db2b16e7e32b976f339b19912ea9e69d58bfb0b2fe6b0e44226529 WatchSource:0}: Error finding container 5754dd89d8db2b16e7e32b976f339b19912ea9e69d58bfb0b2fe6b0e44226529: Status 404 returned error can't find the container with id 5754dd89d8db2b16e7e32b976f339b19912ea9e69d58bfb0b2fe6b0e44226529
Sep 15 07:48:35.936854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.936810    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:35.936946 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.936910    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:35.936984 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:35.936949    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:36.812404 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:36.812360    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerStarted","Data":"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"}
Sep 15 07:48:36.812847 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:36.812430    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerStarted","Data":"5754dd89d8db2b16e7e32b976f339b19912ea9e69d58bfb0b2fe6b0e44226529"}
Sep 15 07:48:36.835262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:36.835200    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podStartSLOduration=1.835182068 podStartE2EDuration="1.835182068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:36.833003383 +0000 UTC m=+1804.927169656" watchObservedRunningTime="2026-09-15 07:48:36.835182068 +0000 UTC m=+1804.929348743"
Sep 15 07:48:37.626079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:37.626038    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:37.631419 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:37.631388    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:37.818760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:37.818717    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:37.819777 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:37.819756    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:48:40.333886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.333847    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:48:40.337998 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.337969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.340674 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.340646    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1755f7-serving-cert\""
Sep 15 07:48:40.349818 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.349787    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:48:40.399634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.396496    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwtp\" (UniqueName: \"kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.399634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.396572    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.399634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.396709    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.399634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.396788    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.498811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.498767    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.498999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.498845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.498999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.498881    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwtp\" (UniqueName: \"kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.498999 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.498917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.498999 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:48:40.498941    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-serving-cert: secret "maas-api-e2e-api-b-1755f7-serving-cert" not found
Sep 15 07:48:40.499184 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:48:40.499030    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls podName:a49613db-74e6-46b5-9bfd-a887d8cc622a nodeName:}" failed. No retries permitted until 2026-09-15 07:48:40.99901121 +0000 UTC m=+1809.093177470 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls") pod "maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" (UID: "a49613db-74e6-46b5-9bfd-a887d8cc622a") : secret "maas-api-e2e-api-b-1755f7-serving-cert" not found
Sep 15 07:48:40.499688 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.499664    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.501485 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.501455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:40.516825 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:40.516792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwtp\" (UniqueName: \"kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:41.004683 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:41.004637    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:41.007300 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:41.007269    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") pod \"maas-api-e2e-api-b-1755f7-78d447c659-pjpwl\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:41.257723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:41.257674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:41.538474 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:41.538442    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:48:41.540085 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:48:41.540052    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49613db_74e6_46b5_9bfd_a887d8cc622a.slice/crio-e6bfd15f234e248c279f07021f1edfd8b09321fa81915cb2b06550e87e3c94f7 WatchSource:0}: Error finding container e6bfd15f234e248c279f07021f1edfd8b09321fa81915cb2b06550e87e3c94f7: Status 404 returned error can't find the container with id e6bfd15f234e248c279f07021f1edfd8b09321fa81915cb2b06550e87e3c94f7
Sep 15 07:48:41.846262 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:41.846223    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" event={"ID":"a49613db-74e6-46b5-9bfd-a887d8cc622a","Type":"ContainerStarted","Data":"e6bfd15f234e248c279f07021f1edfd8b09321fa81915cb2b06550e87e3c94f7"}
Sep 15 07:48:42.860477 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:42.860426    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" event={"ID":"a49613db-74e6-46b5-9bfd-a887d8cc622a","Type":"ContainerStarted","Data":"54390f187c932a91a90f644ccc99585761e361426eecc481fdf4b6f299e81462"}
Sep 15 07:48:42.860969 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:42.860582    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:42.885179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:42.885106    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" podStartSLOduration=2.603270253 podStartE2EDuration="2.885088126s" totalImagesPullingTime="281.817873ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:48:42.881544295 +0000 UTC m=+1810.975710558" watchObservedRunningTime="2026-09-15 07:48:42.885088126 +0000 UTC m=+1810.979254424"
Sep 15 07:48:48.877971 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:48.877934    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:48:53.603722 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.603684    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:53.604175 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.603743    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:48:53.941964 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.941926    2574 generic.go:356] "Generic (PLEG): container finished" podID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerID="a3076cab4494d8d0a5e00a0e9dfe48250007fdfffe5d02e4b4a9980ee7d8d57b" exitCode=0
Sep 15 07:48:53.942219 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.942008    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerDied","Data":"a3076cab4494d8d0a5e00a0e9dfe48250007fdfffe5d02e4b4a9980ee7d8d57b"}
Sep 15 07:48:53.942539 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.942520    2574 scope.go:117] "RemoveContainer" containerID="a3076cab4494d8d0a5e00a0e9dfe48250007fdfffe5d02e4b4a9980ee7d8d57b"
Sep 15 07:48:53.943794 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.943765    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:53.943868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.943835    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:53.943907 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:53.943876    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:48:54.952532 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:54.952481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerStarted","Data":"0aa9c0c9f3e1807f12fa2d117c3552f29ca680df11391df270b7cc86aed8c330"}
Sep 15 07:48:55.335382 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:55.335283    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:48:55.335582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:55.335505    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:55.335658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:55.335571    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:48:56.335661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:56.335614    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:56.336086 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:56.335684    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:48:57.335688 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:57.335647    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:57.336101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:57.335718    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:48:58.336272 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:58.336226    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:58.336713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:58.336288    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:48:59.335791 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:59.335750    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Sep 15 07:48:59.335982 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:48:59.335814    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Sep 15 07:49:00.340381 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:00.340338    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:49:00.340786 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:00.340748    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:49:00.341445 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:00.341423    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:49:04.125409 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:04.125360    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:49:16.102537 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.102497    2574 generic.go:356] "Generic (PLEG): container finished" podID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerID="13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6" exitCode=0
Sep 15 07:49:16.103006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.102578    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerDied","Data":"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"}
Sep 15 07:49:16.103140 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.103102    2574 scope.go:117] "RemoveContainer" containerID="13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"
Sep 15 07:49:16.104366 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.104336    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:49:16.104481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.104423    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:49:16.104481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:16.104473    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:49:17.114551 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:17.114505    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerStarted","Data":"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"}
Sep 15 07:49:17.626282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:17.626240    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:17.626569 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:17.626545    2574 patch_prober.go:28] interesting pod/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 07:49:17.626630 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:17.626610    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 07:49:18.626689 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:18.626644    2574 patch_prober.go:28] interesting pod/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 07:49:18.627156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:18.626721    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 07:49:19.626779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:19.626741    2574 patch_prober.go:28] interesting pod/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 07:49:19.627219 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:19.626816    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 07:49:20.627055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:20.627012    2574 patch_prober.go:28] interesting pod/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 07:49:20.627573 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:20.627076    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 07:49:21.627012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:21.626972    2574 patch_prober.go:28] interesting pod/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused" start-of-body=
Sep 15 07:49:21.627239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:21.627035    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.83:15021/healthz/ready\": dial tcp 10.133.0.83:15021: connect: connection refused"
Sep 15 07:49:22.630781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:22.630749    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:22.631198 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:22.631034    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:22.631731 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:22.631713    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:22.741384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:22.741345    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:49:22.741615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:22.741580    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" podUID="22c8beb6-113e-4c39-8252-85b944ded485" containerName="maas-api" containerID="cri-o://f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44" gracePeriod=30
Sep 15 07:49:23.013232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.013198    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:49:23.156649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.156597    2574 generic.go:356] "Generic (PLEG): container finished" podID="22c8beb6-113e-4c39-8252-85b944ded485" containerID="f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44" exitCode=0
Sep 15 07:49:23.156928 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.156907    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"
Sep 15 07:49:23.157496 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.157473    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" event={"ID":"22c8beb6-113e-4c39-8252-85b944ded485","Type":"ContainerDied","Data":"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"}
Sep 15 07:49:23.157679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.157660    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2" event={"ID":"22c8beb6-113e-4c39-8252-85b944ded485","Type":"ContainerDied","Data":"a484d250c9fd232efaf78835b047816a9cc937bdff8763ba745d0b5f1ec8688f"}
Sep 15 07:49:23.157811 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.157798    2574 scope.go:117] "RemoveContainer" containerID="f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"
Sep 15 07:49:23.159475 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.159232    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs\") pod \"22c8beb6-113e-4c39-8252-85b944ded485\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") "
Sep 15 07:49:23.159475 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.159294    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca\") pod \"22c8beb6-113e-4c39-8252-85b944ded485\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") "
Sep 15 07:49:23.159475 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.159340    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nc5c\" (UniqueName: \"kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c\") pod \"22c8beb6-113e-4c39-8252-85b944ded485\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") "
Sep 15 07:49:23.159475 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.159376    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls\") pod \"22c8beb6-113e-4c39-8252-85b944ded485\" (UID: \"22c8beb6-113e-4c39-8252-85b944ded485\") "
Sep 15 07:49:23.160443 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.160411    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22c8beb6-113e-4c39-8252-85b944ded485" (UID: "22c8beb6-113e-4c39-8252-85b944ded485"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:23.177265 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.176769    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22c8beb6-113e-4c39-8252-85b944ded485" (UID: "22c8beb6-113e-4c39-8252-85b944ded485"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:49:23.177265 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.176910    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22c8beb6-113e-4c39-8252-85b944ded485" (UID: "22c8beb6-113e-4c39-8252-85b944ded485"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:49:23.179226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.179155    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c" (OuterVolumeSpecName: "kube-api-access-4nc5c") pod "22c8beb6-113e-4c39-8252-85b944ded485" (UID: "22c8beb6-113e-4c39-8252-85b944ded485"). InnerVolumeSpecName "kube-api-access-4nc5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:23.197778 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.197749    2574 scope.go:117] "RemoveContainer" containerID="f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"
Sep 15 07:49:23.198157 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:49:23.198128    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44\": container with ID starting with f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44 not found: ID does not exist" containerID="f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"
Sep 15 07:49:23.198239 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.198166    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44"} err="failed to get container status \"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44\": rpc error: code = NotFound desc = could not find container \"f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44\": container with ID starting with f7eb3fcce8b2db586cc1508e985bf01442a8b3dbe1fae004d369572e72f67a44 not found: ID does not exist"
Sep 15 07:49:23.261586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.261492    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:23.261586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.261529    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c8beb6-113e-4c39-8252-85b944ded485-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:23.261586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.261544    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nc5c\" (UniqueName: \"kubernetes.io/projected/22c8beb6-113e-4c39-8252-85b944ded485-kube-api-access-4nc5c\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:23.261586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.261558    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c8beb6-113e-4c39-8252-85b944ded485-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:23.486836 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.486803    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:49:23.489553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:23.489525    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-4ef86b-744b754d6b-br5n2"]
Sep 15 07:49:24.379303 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:24.379264    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c8beb6-113e-4c39-8252-85b944ded485" path="/var/lib/kubelet/pods/22c8beb6-113e-4c39-8252-85b944ded485/volumes"
Sep 15 07:49:30.052753 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:30.052707    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:49:30.053232 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:30.052985    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" containerID="cri-o://0aa9c0c9f3e1807f12fa2d117c3552f29ca680df11391df270b7cc86aed8c330" gracePeriod=30
Sep 15 07:49:30.343226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:30.343112    2574 patch_prober.go:28] interesting pod/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 07:49:30.343226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:30.343198    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:49:30.924055 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:30.924014    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:49:35.236133 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.236085    2574 generic.go:356] "Generic (PLEG): container finished" podID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerID="0aa9c0c9f3e1807f12fa2d117c3552f29ca680df11391df270b7cc86aed8c330" exitCode=0
Sep 15 07:49:35.236553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.236151    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerDied","Data":"0aa9c0c9f3e1807f12fa2d117c3552f29ca680df11391df270b7cc86aed8c330"}
Sep 15 07:49:35.236553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.236236    2574 scope.go:117] "RemoveContainer" containerID="a3076cab4494d8d0a5e00a0e9dfe48250007fdfffe5d02e4b4a9980ee7d8d57b"
Sep 15 07:49:35.349135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.349094    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:49:35.388672 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388637    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.388870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388687    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.388870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388716    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.388870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388759    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dwjdj\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.388870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388776    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.389108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.388797    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.389108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389082    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.389108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389087    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:35.389321 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389203    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.389321 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389247    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo\") pod \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\" (UID: \"bd91a6cd-a760-4ed7-bdc0-b475c791c621\") "
Sep 15 07:49:35.389321 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389275    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389500    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389514    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389632    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389652    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389668    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389682    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.389773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.389688    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:35.391728 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.391689    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj" (OuterVolumeSpecName: "kube-api-access-dwjdj") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "kube-api-access-dwjdj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:35.391937 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.391906    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:35.392165 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.392108    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:35.392252 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.392234    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd91a6cd-a760-4ed7-bdc0-b475c791c621" (UID: "bd91a6cd-a760-4ed7-bdc0-b475c791c621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:49:35.490812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.490769    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjdj\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-kube-api-access-dwjdj\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.490812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.490813    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.491048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.490827    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.491048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.490839    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:35.491048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:35.490849    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd91a6cd-a760-4ed7-bdc0-b475c791c621-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:36.243557 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.243523    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"
Sep 15 07:49:36.243557 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.243548    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z" event={"ID":"bd91a6cd-a760-4ed7-bdc0-b475c791c621","Type":"ContainerDied","Data":"d9e11957d443f04d41718e7a529cf58855a8e5c8a92ba71671a2bac727a61904"}
Sep 15 07:49:36.244050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.243639    2574 scope.go:117] "RemoveContainer" containerID="0aa9c0c9f3e1807f12fa2d117c3552f29ca680df11391df270b7cc86aed8c330"
Sep 15 07:49:36.269185 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.269147    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:49:36.272342 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.272313    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4ef86b-openshift-default-5d9d47d5b6-59v5z"]
Sep 15 07:49:36.380537 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:36.380501    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" path="/var/lib/kubelet/pods/bd91a6cd-a760-4ed7-bdc0-b475c791c621/volumes"
Sep 15 07:49:46.185795 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.185756    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:49:46.186309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.186003    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" podUID="a49613db-74e6-46b5-9bfd-a887d8cc622a" containerName="maas-api" containerID="cri-o://54390f187c932a91a90f644ccc99585761e361426eecc481fdf4b6f299e81462" gracePeriod=30
Sep 15 07:49:46.309789 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.309749    2574 generic.go:356] "Generic (PLEG): container finished" podID="a49613db-74e6-46b5-9bfd-a887d8cc622a" containerID="54390f187c932a91a90f644ccc99585761e361426eecc481fdf4b6f299e81462" exitCode=0
Sep 15 07:49:46.309956 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.309811    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" event={"ID":"a49613db-74e6-46b5-9bfd-a887d8cc622a","Type":"ContainerDied","Data":"54390f187c932a91a90f644ccc99585761e361426eecc481fdf4b6f299e81462"}
Sep 15 07:49:46.467490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.467166    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:49:46.611751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.611714    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llwtp\" (UniqueName: \"kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp\") pod \"a49613db-74e6-46b5-9bfd-a887d8cc622a\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") "
Sep 15 07:49:46.612006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.611985    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca\") pod \"a49613db-74e6-46b5-9bfd-a887d8cc622a\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") "
Sep 15 07:49:46.612216 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.612193    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs\") pod \"a49613db-74e6-46b5-9bfd-a887d8cc622a\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") "
Sep 15 07:49:46.612436 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.612417    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") pod \"a49613db-74e6-46b5-9bfd-a887d8cc622a\" (UID: \"a49613db-74e6-46b5-9bfd-a887d8cc622a\") "
Sep 15 07:49:46.612878 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.612845    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a49613db-74e6-46b5-9bfd-a887d8cc622a" (UID: "a49613db-74e6-46b5-9bfd-a887d8cc622a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:46.613101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.613068    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49613db-74e6-46b5-9bfd-a887d8cc622a-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.616247 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.616190    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a49613db-74e6-46b5-9bfd-a887d8cc622a" (UID: "a49613db-74e6-46b5-9bfd-a887d8cc622a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:49:46.616626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.616498    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp" (OuterVolumeSpecName: "kube-api-access-llwtp") pod "a49613db-74e6-46b5-9bfd-a887d8cc622a" (UID: "a49613db-74e6-46b5-9bfd-a887d8cc622a"). InnerVolumeSpecName "kube-api-access-llwtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:46.616626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.616581    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a49613db-74e6-46b5-9bfd-a887d8cc622a" (UID: "a49613db-74e6-46b5-9bfd-a887d8cc622a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:49:46.714411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.714307    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.714411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.714345    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llwtp\" (UniqueName: \"kubernetes.io/projected/a49613db-74e6-46b5-9bfd-a887d8cc622a-kube-api-access-llwtp\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:46.714411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:46.714355    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a49613db-74e6-46b5-9bfd-a887d8cc622a-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:47.318275 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:47.318241    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"
Sep 15 07:49:47.318677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:47.318243    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl" event={"ID":"a49613db-74e6-46b5-9bfd-a887d8cc622a","Type":"ContainerDied","Data":"e6bfd15f234e248c279f07021f1edfd8b09321fa81915cb2b06550e87e3c94f7"}
Sep 15 07:49:47.318677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:47.318403    2574 scope.go:117] "RemoveContainer" containerID="54390f187c932a91a90f644ccc99585761e361426eecc481fdf4b6f299e81462"
Sep 15 07:49:47.343217 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:47.343181    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:49:47.347039 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:47.347008    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1755f7-78d447c659-pjpwl"]
Sep 15 07:49:48.380047 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:48.379996    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a49613db-74e6-46b5-9bfd-a887d8cc622a" path="/var/lib/kubelet/pods/a49613db-74e6-46b5-9bfd-a887d8cc622a/volumes"
Sep 15 07:49:53.118391 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:53.118354    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:49:53.118818 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:53.118586    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy" containerID="cri-o://c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99" gracePeriod=30
Sep 15 07:49:58.382431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.382350    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:58.390546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.390510    2574 generic.go:356] "Generic (PLEG): container finished" podID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerID="c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99" exitCode=0
Sep 15 07:49:58.390708 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.390561    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"
Sep 15 07:49:58.390708 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.390594    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerDied","Data":"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"}
Sep 15 07:49:58.390708 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.390645    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s" event={"ID":"234a3afc-9483-4f7b-a61f-3d3762322fd6","Type":"ContainerDied","Data":"5754dd89d8db2b16e7e32b976f339b19912ea9e69d58bfb0b2fe6b0e44226529"}
Sep 15 07:49:58.390708 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.390670    2574 scope.go:117] "RemoveContainer" containerID="c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"
Sep 15 07:49:58.400781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.400755    2574 scope.go:117] "RemoveContainer" containerID="13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"
Sep 15 07:49:58.411471 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.411447    2574 scope.go:117] "RemoveContainer" containerID="c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"
Sep 15 07:49:58.411901 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:49:58.411875    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99\": container with ID starting with c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99 not found: ID does not exist" containerID="c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"
Sep 15 07:49:58.411995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.411909    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99"} err="failed to get container status \"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99\": rpc error: code = NotFound desc = could not find container \"c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99\": container with ID starting with c932b7b26df127e88e08b6f369c43a7f9674f06109dea614bfa4a21c4404ae99 not found: ID does not exist"
Sep 15 07:49:58.411995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.411935    2574 scope.go:117] "RemoveContainer" containerID="13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"
Sep 15 07:49:58.412389 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:49:58.412314    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6\": container with ID starting with 13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6 not found: ID does not exist" containerID="13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"
Sep 15 07:49:58.412584 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.412430    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6"} err="failed to get container status \"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6\": rpc error: code = NotFound desc = could not find container \"13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6\": container with ID starting with 13c453871aaa23aff14a95517c98a3827a2d810b212b499e0b90e185046dcae6 not found: ID does not exist"
Sep 15 07:49:58.532318 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532274    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532328    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532369    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532387    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr7d2\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532405    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532420    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532443    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532465    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo\") pod \"234a3afc-9483-4f7b-a61f-3d3762322fd6\" (UID: \"234a3afc-9483-4f7b-a61f-3d3762322fd6\") "
Sep 15 07:49:58.532938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532783    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:58.532938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532845    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data" (OuterVolumeSpecName: "istio-data") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:58.532938 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.532886    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:58.533069 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533014    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:58.533173 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533146    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:49:58.533322 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533302    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.533412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533327    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.533412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533342    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.533412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533356    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.533412 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.533370    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/234a3afc-9483-4f7b-a61f-3d3762322fd6-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.534825 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.534796    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token" (OuterVolumeSpecName: "istio-token") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:58.534825 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.534804    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:49:58.534957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.534822    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:49:58.534957 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.534838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2" (OuterVolumeSpecName: "kube-api-access-vr7d2") pod "234a3afc-9483-4f7b-a61f-3d3762322fd6" (UID: "234a3afc-9483-4f7b-a61f-3d3762322fd6"). InnerVolumeSpecName "kube-api-access-vr7d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:49:58.634978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.634882    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.634978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.634917    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr7d2\" (UniqueName: \"kubernetes.io/projected/234a3afc-9483-4f7b-a61f-3d3762322fd6-kube-api-access-vr7d2\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.634978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.634928    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.634978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.634937    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/234a3afc-9483-4f7b-a61f-3d3762322fd6-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:49:58.715085 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.715044    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:49:58.718743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:58.718691    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1755f7-openshift-default-84d75f88dd-rzb5s"]
Sep 15 07:49:59.807092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:49:59.807049    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:50:00.380661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:00.380617    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" path="/var/lib/kubelet/pods/234a3afc-9483-4f7b-a61f-3d3762322fd6/volumes"
Sep 15 07:50:04.064308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.063055    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:50:04.064308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.063371    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" podUID="818dd830-4abd-44a0-aab0-24a93e2ff59b" containerName="maas-api" containerID="cri-o://925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5" gracePeriod=30
Sep 15 07:50:04.335049 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.335020    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:50:04.392214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.392175    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdmz4\" (UniqueName: \"kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4\") pod \"818dd830-4abd-44a0-aab0-24a93e2ff59b\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") "
Sep 15 07:50:04.392417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.392225    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs\") pod \"818dd830-4abd-44a0-aab0-24a93e2ff59b\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") "
Sep 15 07:50:04.392417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.392246    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") pod \"818dd830-4abd-44a0-aab0-24a93e2ff59b\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") "
Sep 15 07:50:04.394676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.394635    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "818dd830-4abd-44a0-aab0-24a93e2ff59b" (UID: "818dd830-4abd-44a0-aab0-24a93e2ff59b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:50:04.394676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.394648    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "818dd830-4abd-44a0-aab0-24a93e2ff59b" (UID: "818dd830-4abd-44a0-aab0-24a93e2ff59b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:50:04.394869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.394841    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4" (OuterVolumeSpecName: "kube-api-access-mdmz4") pod "818dd830-4abd-44a0-aab0-24a93e2ff59b" (UID: "818dd830-4abd-44a0-aab0-24a93e2ff59b"). InnerVolumeSpecName "kube-api-access-mdmz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:04.430989 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.430946    2574 generic.go:356] "Generic (PLEG): container finished" podID="818dd830-4abd-44a0-aab0-24a93e2ff59b" containerID="925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5" exitCode=0
Sep 15 07:50:04.431213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.430991    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"
Sep 15 07:50:04.431213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.431024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" event={"ID":"818dd830-4abd-44a0-aab0-24a93e2ff59b","Type":"ContainerDied","Data":"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"}
Sep 15 07:50:04.431213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.431078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h" event={"ID":"818dd830-4abd-44a0-aab0-24a93e2ff59b","Type":"ContainerDied","Data":"229ad46a602bc2b6e283f5ac837a79a685c448549f1451738a379aecbe2f6666"}
Sep 15 07:50:04.431213 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.431101    2574 scope.go:117] "RemoveContainer" containerID="925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"
Sep 15 07:50:04.443400 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.443376    2574 scope.go:117] "RemoveContainer" containerID="925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"
Sep 15 07:50:04.443784 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:50:04.443759    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5\": container with ID starting with 925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5 not found: ID does not exist" containerID="925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"
Sep 15 07:50:04.443842 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.443795    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5"} err="failed to get container status \"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5\": rpc error: code = NotFound desc = could not find container \"925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5\": container with ID starting with 925c286d2d8407c865065f81e25bb6ff1a8b184fca498d1779fc29709c4f0ae5 not found: ID does not exist"
Sep 15 07:50:04.494276 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.494237    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca\") pod \"818dd830-4abd-44a0-aab0-24a93e2ff59b\" (UID: \"818dd830-4abd-44a0-aab0-24a93e2ff59b\") "
Sep 15 07:50:04.494555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.494535    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdmz4\" (UniqueName: \"kubernetes.io/projected/818dd830-4abd-44a0-aab0-24a93e2ff59b-kube-api-access-mdmz4\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:04.494650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.494558    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:04.494650 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.494572    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/818dd830-4abd-44a0-aab0-24a93e2ff59b-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:04.494762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.494687    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "818dd830-4abd-44a0-aab0-24a93e2ff59b" (UID: "818dd830-4abd-44a0-aab0-24a93e2ff59b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:50:04.596062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.595967    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/818dd830-4abd-44a0-aab0-24a93e2ff59b-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:04.753898 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.753860    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:50:04.758324 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:04.758290    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-3a4c9f-596d8b9674-hwn8h"]
Sep 15 07:50:06.379473 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:06.379435    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="818dd830-4abd-44a0-aab0-24a93e2ff59b" path="/var/lib/kubelet/pods/818dd830-4abd-44a0-aab0-24a93e2ff59b/volumes"
Sep 15 07:50:22.170131 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.170074    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:50:22.170666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.170426    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" podUID="82cf8264-b253-4ff9-867f-9391f08f09d0" containerName="maas-api" containerID="cri-o://5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4" gracePeriod=30
Sep 15 07:50:22.429611 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.429540    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:50:22.545070 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.545022    2574 generic.go:356] "Generic (PLEG): container finished" podID="82cf8264-b253-4ff9-867f-9391f08f09d0" containerID="5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4" exitCode=0
Sep 15 07:50:22.545268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.545161    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" event={"ID":"82cf8264-b253-4ff9-867f-9391f08f09d0","Type":"ContainerDied","Data":"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"}
Sep 15 07:50:22.545268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.545244    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx" event={"ID":"82cf8264-b253-4ff9-867f-9391f08f09d0","Type":"ContainerDied","Data":"6c1874521b39c3f49e358beb697eeda092c9db050351eff89eec890869f268b8"}
Sep 15 07:50:22.545268 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.545177    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"
Sep 15 07:50:22.545380 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.545262    2574 scope.go:117] "RemoveContainer" containerID="5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"
Sep 15 07:50:22.557187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.557157    2574 scope.go:117] "RemoveContainer" containerID="5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"
Sep 15 07:50:22.557636 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:50:22.557586    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4\": container with ID starting with 5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4 not found: ID does not exist" containerID="5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"
Sep 15 07:50:22.557753 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.557630    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4"} err="failed to get container status \"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4\": rpc error: code = NotFound desc = could not find container \"5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4\": container with ID starting with 5a0dcfc381c95aa369ab524618320bb8438762bde025ad740c8b4fe70fb5c3c4 not found: ID does not exist"
Sep 15 07:50:22.581555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.581514    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca\") pod \"82cf8264-b253-4ff9-867f-9391f08f09d0\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") "
Sep 15 07:50:22.581759 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.581605    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs\") pod \"82cf8264-b253-4ff9-867f-9391f08f09d0\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") "
Sep 15 07:50:22.581759 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.581640    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfzzc\" (UniqueName: \"kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc\") pod \"82cf8264-b253-4ff9-867f-9391f08f09d0\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") "
Sep 15 07:50:22.581759 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.581692    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls\") pod \"82cf8264-b253-4ff9-867f-9391f08f09d0\" (UID: \"82cf8264-b253-4ff9-867f-9391f08f09d0\") "
Sep 15 07:50:22.581979 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.581950    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "82cf8264-b253-4ff9-867f-9391f08f09d0" (UID: "82cf8264-b253-4ff9-867f-9391f08f09d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:50:22.584445 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.584405    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "82cf8264-b253-4ff9-867f-9391f08f09d0" (UID: "82cf8264-b253-4ff9-867f-9391f08f09d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:50:22.584445 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.584426    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "82cf8264-b253-4ff9-867f-9391f08f09d0" (UID: "82cf8264-b253-4ff9-867f-9391f08f09d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:50:22.585012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.584985    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc" (OuterVolumeSpecName: "kube-api-access-cfzzc") pod "82cf8264-b253-4ff9-867f-9391f08f09d0" (UID: "82cf8264-b253-4ff9-867f-9391f08f09d0"). InnerVolumeSpecName "kube-api-access-cfzzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:22.682606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.682505    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/82cf8264-b253-4ff9-867f-9391f08f09d0-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.682606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.682545    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.682606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.682558    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfzzc\" (UniqueName: \"kubernetes.io/projected/82cf8264-b253-4ff9-867f-9391f08f09d0-kube-api-access-cfzzc\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.682606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.682570    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/82cf8264-b253-4ff9-867f-9391f08f09d0-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:22.877092 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.877057    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:50:22.879489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:22.879466    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-27b9d8-85dc984d65-87qrx"]
Sep 15 07:50:24.379494 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:24.379457    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82cf8264-b253-4ff9-867f-9391f08f09d0" path="/var/lib/kubelet/pods/82cf8264-b253-4ff9-867f-9391f08f09d0/volumes"
Sep 15 07:50:25.812385 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:25.812345    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:50:28.751101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:28.751056    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:50:28.751609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:28.751299    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy" containerID="cri-o://42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d" gracePeriod=30
Sep 15 07:50:34.008614 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.008572    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:50:34.197349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197310    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197392    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197415    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqx4k\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197515    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197551    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197582    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197609    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197643    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197682    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy\") pod \"9693864f-0777-449a-b4ca-65dee8c3ced6\" (UID: \"9693864f-0777-449a-b4ca-65dee8c3ced6\") "
Sep 15 07:50:34.197850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197689    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:50:34.197850 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197786    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.197900    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.198040    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.198057    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.198071    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.198078    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:50:34.198162 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.198096    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:50:34.199909 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.199868    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:34.200049 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.199912    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:50:34.200049 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.199949    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:50:34.200049 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.199957    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k" (OuterVolumeSpecName: "kube-api-access-dqx4k") pod "9693864f-0777-449a-b4ca-65dee8c3ced6" (UID: "9693864f-0777-449a-b4ca-65dee8c3ced6"). InnerVolumeSpecName "kube-api-access-dqx4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:50:34.299370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299328    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9693864f-0777-449a-b4ca-65dee8c3ced6-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.299370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299364    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.299370 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299375    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.299632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299387    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.299632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299397    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.299632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.299406    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqx4k\" (UniqueName: \"kubernetes.io/projected/9693864f-0777-449a-b4ca-65dee8c3ced6-kube-api-access-dqx4k\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:50:34.623050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.622955    2574 generic.go:356] "Generic (PLEG): container finished" podID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerID="42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d" exitCode=0
Sep 15 07:50:34.623050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.623003    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerDied","Data":"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"}
Sep 15 07:50:34.623050 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.623024    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"
Sep 15 07:50:34.623302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.623091    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg" event={"ID":"9693864f-0777-449a-b4ca-65dee8c3ced6","Type":"ContainerDied","Data":"78393e1459bd501e720f4aeb47ac19a4838849642eb1b7d673658ab022166730"}
Sep 15 07:50:34.623302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.623146    2574 scope.go:117] "RemoveContainer" containerID="42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"
Sep 15 07:50:34.632518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.632491    2574 scope.go:117] "RemoveContainer" containerID="eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"
Sep 15 07:50:34.640624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.640585    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:50:34.641641 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.641612    2574 scope.go:117] "RemoveContainer" containerID="42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"
Sep 15 07:50:34.642030 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:50:34.642001    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d\": container with ID starting with 42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d not found: ID does not exist" containerID="42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"
Sep 15 07:50:34.642030 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.642044    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d"} err="failed to get container status \"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d\": rpc error: code = NotFound desc = could not find container \"42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d\": container with ID starting with 42e968180a71fdc187a6f2cd8c297a8e3fb7db28eb464bcc18b05d363572cb9d not found: ID does not exist"
Sep 15 07:50:34.642266 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.642074    2574 scope.go:117] "RemoveContainer" containerID="eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"
Sep 15 07:50:34.642571 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:50:34.642450    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11\": container with ID starting with eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11 not found: ID does not exist" containerID="eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"
Sep 15 07:50:34.642666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.642592    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11"} err="failed to get container status \"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11\": rpc error: code = NotFound desc = could not find container \"eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11\": container with ID starting with eb41cc5faf82a870b8500ae5a7a79f31eb27a907cad599a54366878683aa4e11 not found: ID does not exist"
Sep 15 07:50:34.645212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:34.645189    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-27b9d8-openshift-default-7dccc89bb4-xs7lg"]
Sep 15 07:50:36.379615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:36.379575    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" path="/var/lib/kubelet/pods/9693864f-0777-449a-b4ca-65dee8c3ced6/volumes"
Sep 15 07:50:48.508648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:48.508591    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:50:50.320675 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:50:50.320625    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:51:01.114763 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.114720    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:51:01.115326 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.114980    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" podUID="e5172b04-3747-4acb-b1dd-3036a519ec9b" containerName="maas-api" containerID="cri-o://beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d" gracePeriod=30
Sep 15 07:51:01.592975 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.592942    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:51:01.653854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.653809    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca\") pod \"e5172b04-3747-4acb-b1dd-3036a519ec9b\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") "
Sep 15 07:51:01.653854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.653859    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") pod \"e5172b04-3747-4acb-b1dd-3036a519ec9b\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") "
Sep 15 07:51:01.654142 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.653887    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs\") pod \"e5172b04-3747-4acb-b1dd-3036a519ec9b\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") "
Sep 15 07:51:01.654142 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.653920    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sckd\" (UniqueName: \"kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd\") pod \"e5172b04-3747-4acb-b1dd-3036a519ec9b\" (UID: \"e5172b04-3747-4acb-b1dd-3036a519ec9b\") "
Sep 15 07:51:01.654623 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.654254    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5172b04-3747-4acb-b1dd-3036a519ec9b" (UID: "e5172b04-3747-4acb-b1dd-3036a519ec9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:51:01.656702 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.656625    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5172b04-3747-4acb-b1dd-3036a519ec9b" (UID: "e5172b04-3747-4acb-b1dd-3036a519ec9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:01.656796 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.656699    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd" (OuterVolumeSpecName: "kube-api-access-5sckd") pod "e5172b04-3747-4acb-b1dd-3036a519ec9b" (UID: "e5172b04-3747-4acb-b1dd-3036a519ec9b"). InnerVolumeSpecName "kube-api-access-5sckd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:01.656796 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.656782    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5172b04-3747-4acb-b1dd-3036a519ec9b" (UID: "e5172b04-3747-4acb-b1dd-3036a519ec9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:01.755679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.755636    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5172b04-3747-4acb-b1dd-3036a519ec9b-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.755679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.755671    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.755679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.755680    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5172b04-3747-4acb-b1dd-3036a519ec9b-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.755679 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.755688    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sckd\" (UniqueName: \"kubernetes.io/projected/e5172b04-3747-4acb-b1dd-3036a519ec9b-kube-api-access-5sckd\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:01.781029 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.780993    2574 generic.go:356] "Generic (PLEG): container finished" podID="e5172b04-3747-4acb-b1dd-3036a519ec9b" containerID="beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d" exitCode=0
Sep 15 07:51:01.781222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.781045    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"
Sep 15 07:51:01.781222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.781070    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" event={"ID":"e5172b04-3747-4acb-b1dd-3036a519ec9b","Type":"ContainerDied","Data":"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"}
Sep 15 07:51:01.781222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.781149    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd" event={"ID":"e5172b04-3747-4acb-b1dd-3036a519ec9b","Type":"ContainerDied","Data":"1864eab42477585a83e3855e7391c7be4728ad119a3e63b46234ad9b2960bc41"}
Sep 15 07:51:01.781222 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.781180    2574 scope.go:117] "RemoveContainer" containerID="beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"
Sep 15 07:51:01.793346 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.793289    2574 scope.go:117] "RemoveContainer" containerID="beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"
Sep 15 07:51:01.793713 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:51:01.793691    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d\": container with ID starting with beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d not found: ID does not exist" containerID="beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"
Sep 15 07:51:01.793761 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.793725    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d"} err="failed to get container status \"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d\": rpc error: code = NotFound desc = could not find container \"beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d\": container with ID starting with beb40e774245762f38c2ebdb53759f33e9716e7e3334ecae7d6845a3e560753d not found: ID does not exist"
Sep 15 07:51:01.816329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.816288    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:51:01.826099 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:01.826062    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f56165-556cc464bb-msmsd"]
Sep 15 07:51:02.379197 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:02.379161    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5172b04-3747-4acb-b1dd-3036a519ec9b" path="/var/lib/kubelet/pods/e5172b04-3747-4acb-b1dd-3036a519ec9b/volumes"
Sep 15 07:51:05.340867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.340823    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" podUID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" containerName="main" containerID="cri-o://e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db" gracePeriod=45
Sep 15 07:51:05.613986 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.613721    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:51:05.695254 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695219    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695264    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695299    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695325    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695358    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54zh8\" (UniqueName: \"kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695425    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs\") pod \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\" (UID: \"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632\") "
Sep 15 07:51:05.695718 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695549    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.695718 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695588    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home" (OuterVolumeSpecName: "home") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.695718 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695627    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache" (OuterVolumeSpecName: "model-cache") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.695869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695817    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.695869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695839    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.695869 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.695855    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.697619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.697592    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:51:05.697720 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.697682    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8" (OuterVolumeSpecName: "kube-api-access-54zh8") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "kube-api-access-54zh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:05.697720 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.697696    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm" (OuterVolumeSpecName: "dshm") pod "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" (UID: "b34e7a04-8a74-4f74-bd7b-9d0a6bb17632"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:05.796887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.796830    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.796887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.796881    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54zh8\" (UniqueName: \"kubernetes.io/projected/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-kube-api-access-54zh8\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.796887 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.796893    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:05.806866 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.806832    2574 generic.go:356] "Generic (PLEG): container finished" podID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" containerID="e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db" exitCode=0
Sep 15 07:51:05.807053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.806879    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"
Sep 15 07:51:05.807053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.806901    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" event={"ID":"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632","Type":"ContainerDied","Data":"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"}
Sep 15 07:51:05.807053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.806932    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4" event={"ID":"b34e7a04-8a74-4f74-bd7b-9d0a6bb17632","Type":"ContainerDied","Data":"2279130790b1aaf450ec243fa057b3980854e860bd879ad11904af03f8bd9dce"}
Sep 15 07:51:05.807053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.806950    2574 scope.go:117] "RemoveContainer" containerID="e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"
Sep 15 07:51:05.816653 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.816627    2574 scope.go:117] "RemoveContainer" containerID="e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"
Sep 15 07:51:05.817013 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:51:05.816991    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db\": container with ID starting with e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db not found: ID does not exist" containerID="e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"
Sep 15 07:51:05.817070 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.817022    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db"} err="failed to get container status \"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db\": rpc error: code = NotFound desc = could not find container \"e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db\": container with ID starting with e246326a3fb57234e230b4f5bf4984e47cfc664830f13bfc8ddd0d754da427db not found: ID does not exist"
Sep 15 07:51:05.829781 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.829748    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:51:05.831978 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:05.831953    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f56165/premium-w4-f56165-kserve-58df7879-2kcj4"]
Sep 15 07:51:06.379355 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:06.379308    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" path="/var/lib/kubelet/pods/b34e7a04-8a74-4f74-bd7b-9d0a6bb17632/volumes"
Sep 15 07:51:07.722289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:07.722254    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:51:07.722725 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:07.722476    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy" containerID="cri-o://f300d330361acfdd4fb3bd627d472cc564028ee938ac12c78aeee8c29724c5c7" gracePeriod=30
Sep 15 07:51:12.852253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:12.852199    2574 generic.go:356] "Generic (PLEG): container finished" podID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerID="f300d330361acfdd4fb3bd627d472cc564028ee938ac12c78aeee8c29724c5c7" exitCode=0
Sep 15 07:51:12.852710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:12.852281    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerDied","Data":"f300d330361acfdd4fb3bd627d472cc564028ee938ac12c78aeee8c29724c5c7"}
Sep 15 07:51:12.852710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:12.852367    2574 scope.go:117] "RemoveContainer" containerID="35ca8dfb8146b8fdc977efb966b2fd77e4f1a9d79b637ccd86f1e06c82e8852b"
Sep 15 07:51:12.992245 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:12.992167    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:51:13.168494 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168460    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168494 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168504    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168545    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168578    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps82g\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168619    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168650    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.168762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168727    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.169027 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168781    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.169027 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168808    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket\") pod \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\" (UID: \"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100\") "
Sep 15 07:51:13.169027 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168832    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:13.169027 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.168846    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data" (OuterVolumeSpecName: "istio-data") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:13.169279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.169094    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:13.169279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.169167    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:51:13.169279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.169179    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.169279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.169199    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.169512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.169338    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:13.171182 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.171157    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:13.171309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.171233    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:51:13.171362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.171306    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g" (OuterVolumeSpecName: "kube-api-access-ps82g") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "kube-api-access-ps82g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:13.171401 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.171379    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" (UID: "f8a16f67-dcc9-4e8f-a1f2-1f36d8039100"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270316    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270352    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270362    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270390    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps82g\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-kube-api-access-ps82g\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270400    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270408    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.270415 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.270417    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:13.859841 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.859808    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"
Sep 15 07:51:13.860329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.859808    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc" event={"ID":"f8a16f67-dcc9-4e8f-a1f2-1f36d8039100","Type":"ContainerDied","Data":"c4b666eddb2a3c70d52414d306c088680949205faccf862468538742e1e4ddbc"}
Sep 15 07:51:13.860329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.859941    2574 scope.go:117] "RemoveContainer" containerID="f300d330361acfdd4fb3bd627d472cc564028ee938ac12c78aeee8c29724c5c7"
Sep 15 07:51:13.881913 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.881874    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:51:13.885776 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:13.885744    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-f56165-openshift-default-6f8477bdc4-6vshc"]
Sep 15 07:51:14.379541 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:14.379509    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" path="/var/lib/kubelet/pods/f8a16f67-dcc9-4e8f-a1f2-1f36d8039100/volumes"
Sep 15 07:51:14.801047 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:14.801010    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:51:18.030209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030173    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-b0a4e2"]
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030526    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030537    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030547    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818dd830-4abd-44a0-aab0-24a93e2ff59b" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030552    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="818dd830-4abd-44a0-aab0-24a93e2ff59b" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030561    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" containerName="main"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030567    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" containerName="main"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030573    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030578    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030586    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030592    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030599    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82cf8264-b253-4ff9-867f-9391f08f09d0" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030603    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="82cf8264-b253-4ff9-867f-9391f08f09d0" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030616    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030622    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030628    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030634    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030641    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5172b04-3747-4acb-b1dd-3036a519ec9b" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030647    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5172b04-3747-4acb-b1dd-3036a519ec9b" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030655    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030660    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030669    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c8beb6-113e-4c39-8252-85b944ded485" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030674    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c8beb6-113e-4c39-8252-85b944ded485" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030680    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030684    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030691    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a49613db-74e6-46b5-9bfd-a887d8cc622a" containerName="maas-api"
Sep 15 07:51:18.030698 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030695    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a49613db-74e6-46b5-9bfd-a887d8cc622a" containerName="maas-api"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030749    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030756    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b34e7a04-8a74-4f74-bd7b-9d0a6bb17632" containerName="main"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030763    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030769    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030775    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030783    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030788    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a49613db-74e6-46b5-9bfd-a887d8cc622a" containerName="maas-api"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030794    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9693864f-0777-449a-b4ca-65dee8c3ced6" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030800    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="82cf8264-b253-4ff9-867f-9391f08f09d0" containerName="maas-api"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030804    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c8beb6-113e-4c39-8252-85b944ded485" containerName="maas-api"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030810    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="234a3afc-9483-4f7b-a61f-3d3762322fd6" containerName="istio-proxy"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030814    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5172b04-3747-4acb-b1dd-3036a519ec9b" containerName="maas-api"
Sep 15 07:51:18.031504 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.030819    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="818dd830-4abd-44a0-aab0-24a93e2ff59b" containerName="maas-api"
Sep 15 07:51:18.035531 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.035494    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:18.037251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.037217    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-b0a4e2"]
Sep 15 07:51:18.037632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.037613    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8zjf\""
Sep 15 07:51:18.215448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.215406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t9sb\" (UniqueName: \"kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb\") pod \"test-curl-7660-b0a4e2\" (UID: \"c3c32967-1a0c-4fbd-9b98-97be0f603f21\") " pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:18.316551 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.316446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t9sb\" (UniqueName: \"kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb\") pod \"test-curl-7660-b0a4e2\" (UID: \"c3c32967-1a0c-4fbd-9b98-97be0f603f21\") " pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:18.324405 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.324370    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t9sb\" (UniqueName: \"kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb\") pod \"test-curl-7660-b0a4e2\" (UID: \"c3c32967-1a0c-4fbd-9b98-97be0f603f21\") " pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:18.346281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.346243    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:18.619797 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.619770    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-b0a4e2"]
Sep 15 07:51:18.622047 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:51:18.622014    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3c32967_1a0c_4fbd_9b98_97be0f603f21.slice/crio-ccfd2e30a1d5fef346331fe7a7d1d6ca2b0d85ccb665e2701928cd2e61ecc154 WatchSource:0}: Error finding container ccfd2e30a1d5fef346331fe7a7d1d6ca2b0d85ccb665e2701928cd2e61ecc154: Status 404 returned error can't find the container with id ccfd2e30a1d5fef346331fe7a7d1d6ca2b0d85ccb665e2701928cd2e61ecc154
Sep 15 07:51:18.896363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.896260    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-b0a4e2" event={"ID":"c3c32967-1a0c-4fbd-9b98-97be0f603f21","Type":"ContainerStarted","Data":"1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c"}
Sep 15 07:51:18.896363 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.896310    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-b0a4e2" event={"ID":"c3c32967-1a0c-4fbd-9b98-97be0f603f21","Type":"ContainerStarted","Data":"ccfd2e30a1d5fef346331fe7a7d1d6ca2b0d85ccb665e2701928cd2e61ecc154"}
Sep 15 07:51:18.911952 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:18.911889    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7660-b0a4e2" podStartSLOduration=0.911870735 podStartE2EDuration="911.870735ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:51:18.909577286 +0000 UTC m=+1967.003743560" watchObservedRunningTime="2026-09-15 07:51:18.911870735 +0000 UTC m=+1967.006036997"
Sep 15 07:51:19.476626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:19.476582    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-b0a4e2"]
Sep 15 07:51:19.479954 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:19.479920    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-b0a4e2"]
Sep 15 07:51:19.904146 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:51:19.903790    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7660-b0a4e2" container="test-curl-7660-b0a4e2"
Sep 15 07:51:20.910546 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:20.910479    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7660-b0a4e2" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" containerName="test-curl-7660-b0a4e2" containerID="cri-o://1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c" gracePeriod=2
Sep 15 07:51:23.102382 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.102350    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:23.161190 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.161151    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t9sb\" (UniqueName: \"kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb\") pod \"c3c32967-1a0c-4fbd-9b98-97be0f603f21\" (UID: \"c3c32967-1a0c-4fbd-9b98-97be0f603f21\") "
Sep 15 07:51:23.163446 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.163416    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb" (OuterVolumeSpecName: "kube-api-access-8t9sb") pod "c3c32967-1a0c-4fbd-9b98-97be0f603f21" (UID: "c3c32967-1a0c-4fbd-9b98-97be0f603f21"). InnerVolumeSpecName "kube-api-access-8t9sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:51:23.262228 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.262193    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t9sb\" (UniqueName: \"kubernetes.io/projected/c3c32967-1a0c-4fbd-9b98-97be0f603f21-kube-api-access-8t9sb\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:51:23.929044 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.929005    2574 generic.go:356] "Generic (PLEG): container finished" podID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" containerID="1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c" exitCode=137
Sep 15 07:51:23.929044 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.929045    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:23.929309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.929101    2574 scope.go:117] "RemoveContainer" containerID="1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c"
Sep 15 07:51:23.938834 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.938811    2574 scope.go:117] "RemoveContainer" containerID="1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c"
Sep 15 07:51:23.939262 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:51:23.939235    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c\": container with ID starting with 1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c not found: ID does not exist" containerID="1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c"
Sep 15 07:51:23.939262 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:51:23.939245    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-b0a4e2\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" pod="opendatahub/test-curl-7660-b0a4e2"
Sep 15 07:51:23.939423 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:23.939269    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c"} err="failed to get container status \"1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c\": rpc error: code = NotFound desc = could not find container \"1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c\": container with ID starting with 1d7bd3888e53d5e8f4b6cabdccba4380d25760d36c2202a34ad6145e12f1dd1c not found: ID does not exist"
Sep 15 07:51:24.378701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:24.378612    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" path="/var/lib/kubelet/pods/c3c32967-1a0c-4fbd-9b98-97be0f603f21/volumes"
Sep 15 07:51:26.303427 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.303333    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:51:26.893489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.893449    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:51:26.894010 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.893993    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:26.894061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.894014    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a16f67-dcc9-4e8f-a1f2-1f36d8039100" containerName="istio-proxy"
Sep 15 07:51:26.894061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.894041    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" containerName="test-curl-7660-b0a4e2"
Sep 15 07:51:26.894061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.894050    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" containerName="test-curl-7660-b0a4e2"
Sep 15 07:51:26.894195 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.894157    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd91a6cd-a760-4ed7-bdc0-b475c791c621" containerName="istio-proxy"
Sep 15 07:51:26.894195 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.894169    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c32967-1a0c-4fbd-9b98-97be0f603f21" containerName="test-curl-7660-b0a4e2"
Sep 15 07:51:26.898871 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.898846    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.901086 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.901058    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"kube-root-ca.crt\""
Sep 15 07:51:26.901224 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.901142    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"openshift-service-ca.crt\""
Sep 15 07:51:26.901809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.901790    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"sub-test-model-04464b-kserve-self-signed-certs\""
Sep 15 07:51:26.901955 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.901891    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-04464b\"/\"default-dockercfg-79sv4\""
Sep 15 07:51:26.906008 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.905984    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:51:26.998319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998280    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.998319 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998324    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.998586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998375    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.998586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998446    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.998586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998536    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:26.998586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:26.998575    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfzc\" (UniqueName: \"kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.099980 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.099926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.099995    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfzc\" (UniqueName: \"kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100171    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100219    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100404    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100543 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.100649 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.100550    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.102666 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.102634    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.103044 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.103025    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.108804 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.108776    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfzc\" (UniqueName: \"kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc\") pod \"sub-test-model-04464b-kserve-68bf8c68d7-l9477\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") " pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.210657 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.210620    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:27.486678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.486646    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:51:27.488794 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:51:27.488758    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45c2d103_0376_42e2_a044_bae946bbf232.slice/crio-6401ae08afdd226885c90e63939c9508d80936f2ec51131196012a3d7e2ab670 WatchSource:0}: Error finding container 6401ae08afdd226885c90e63939c9508d80936f2ec51131196012a3d7e2ab670: Status 404 returned error can't find the container with id 6401ae08afdd226885c90e63939c9508d80936f2ec51131196012a3d7e2ab670
Sep 15 07:51:27.957302 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.957263    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" event={"ID":"45c2d103-0376-42e2-a044-bae946bbf232","Type":"ContainerStarted","Data":"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"}
Sep 15 07:51:27.957631 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.957313    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" event={"ID":"45c2d103-0376-42e2-a044-bae946bbf232","Type":"ContainerStarted","Data":"6401ae08afdd226885c90e63939c9508d80936f2ec51131196012a3d7e2ab670"}
Sep 15 07:51:27.975441 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:27.975377    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" podStartSLOduration=1.9753589809999998 podStartE2EDuration="1.975358981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:51:27.973009764 +0000 UTC m=+1976.067176027" watchObservedRunningTime="2026-09-15 07:51:27.975358981 +0000 UTC m=+1976.069525242"
Sep 15 07:51:35.902219 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:35.902170    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:51:37.210940 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:37.210888    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:37.210940 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:37.210957    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:37.223915 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:37.223885    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:38.024459 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:38.024426    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:51:43.508457 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.508415    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:51:43.512552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.512521    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.515492 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.515463    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"kube-root-ca.crt\""
Sep 15 07:51:43.515634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.515501    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"openshift-service-ca.crt\""
Sep 15 07:51:43.515634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.515532    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"sub-test-model-e3a482-kserve-self-signed-certs\""
Sep 15 07:51:43.515634 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.515585    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"default-dockercfg-jgj2b\""
Sep 15 07:51:43.522683 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.522657    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:51:43.659670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659624    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.659670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659673    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.659893 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659780    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.659893 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659833    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.659893 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659879    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.659997 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.659921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbln\" (UniqueName: \"kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761041 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.760949    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761041 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.760999    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761041 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761051    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761088    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbln\" (UniqueName: \"kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761362 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761175    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761554 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761508    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761554 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761527    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.761885 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.761845    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.763752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.763729    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.763954 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.763925    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.769081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.769053    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbln\" (UniqueName: \"kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln\") pod \"sub-test-model-e3a482-kserve-5c867f845c-fcgpq\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:43.823106 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:43.823068    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:44.105209 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:44.105179    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:51:44.106829 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:51:44.106799    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07a64b16_4380_4768_a935_b5272005e998.slice/crio-723e1f9f10653335433bf05fb034dd77c05c018020e77fbca84592df46c6ba3b WatchSource:0}: Error finding container 723e1f9f10653335433bf05fb034dd77c05c018020e77fbca84592df46c6ba3b: Status 404 returned error can't find the container with id 723e1f9f10653335433bf05fb034dd77c05c018020e77fbca84592df46c6ba3b
Sep 15 07:51:45.061961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:45.061914    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" event={"ID":"07a64b16-4380-4768-a935-b5272005e998","Type":"ContainerStarted","Data":"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"}
Sep 15 07:51:45.061961 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:45.061968    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" event={"ID":"07a64b16-4380-4768-a935-b5272005e998","Type":"ContainerStarted","Data":"723e1f9f10653335433bf05fb034dd77c05c018020e77fbca84592df46c6ba3b"}
Sep 15 07:51:45.082677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:45.082618    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" podStartSLOduration=2.082601419 podStartE2EDuration="2.082601419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:51:45.080177884 +0000 UTC m=+1993.174344147" watchObservedRunningTime="2026-09-15 07:51:45.082601419 +0000 UTC m=+1993.176767709"
Sep 15 07:51:47.210237 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:47.210191    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:51:53.823425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:53.823373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:53.823425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:53.823434    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:53.835936 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:53.835899    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:51:54.124368 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:51:54.124269    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:52:10.412376 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:10.412335    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:52:20.507031 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:20.506992    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:52:27.022448 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.022407    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7660-86910b"]
Sep 15 07:52:27.026143 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.026108    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:27.028552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.028520    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-x8zjf\""
Sep 15 07:52:27.030697 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.030670    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-86910b"]
Sep 15 07:52:27.150317 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.150270    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzvt\" (UniqueName: \"kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt\") pod \"test-curl-7660-86910b\" (UID: \"872924e5-c0b9-4d01-b082-0880c9940e80\") " pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:27.251089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.251034    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzvt\" (UniqueName: \"kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt\") pod \"test-curl-7660-86910b\" (UID: \"872924e5-c0b9-4d01-b082-0880c9940e80\") " pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:27.258943 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.258904    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzvt\" (UniqueName: \"kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt\") pod \"test-curl-7660-86910b\" (UID: \"872924e5-c0b9-4d01-b082-0880c9940e80\") " pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:27.337762 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.337667    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:27.610098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:27.610070    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7660-86910b"]
Sep 15 07:52:27.612360 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:52:27.612326    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod872924e5_c0b9_4d01_b082_0880c9940e80.slice/crio-d0259ad408cb3de28d3d363694c2d73e4d40a4258d508056000df95be835a588 WatchSource:0}: Error finding container d0259ad408cb3de28d3d363694c2d73e4d40a4258d508056000df95be835a588: Status 404 returned error can't find the container with id d0259ad408cb3de28d3d363694c2d73e4d40a4258d508056000df95be835a588
Sep 15 07:52:28.302864 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:28.302826    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-86910b" event={"ID":"872924e5-c0b9-4d01-b082-0880c9940e80","Type":"ContainerStarted","Data":"06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72"}
Sep 15 07:52:28.302864 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:28.302868    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7660-86910b" event={"ID":"872924e5-c0b9-4d01-b082-0880c9940e80","Type":"ContainerStarted","Data":"d0259ad408cb3de28d3d363694c2d73e4d40a4258d508056000df95be835a588"}
Sep 15 07:52:28.317230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:28.317173    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7660-86910b" podStartSLOduration=1.317154116 podStartE2EDuration="1.317154116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:52:28.316281821 +0000 UTC m=+2036.410448083" watchObservedRunningTime="2026-09-15 07:52:28.317154116 +0000 UTC m=+2036.411320379"
Sep 15 07:52:28.881882 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:28.881834    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7660-86910b"]
Sep 15 07:52:28.888658 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:28.888608    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7660-86910b"]
Sep 15 07:52:29.308531 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:29.308495    2574 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7660-86910b" container="test-curl-7660-86910b"
Sep 15 07:52:30.313664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:30.313613    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7660-86910b" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" containerName="test-curl-7660-86910b" containerID="cri-o://06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72" gracePeriod=2
Sep 15 07:52:30.315704 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:30.315672    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-86910b\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:30.900340 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:30.900304    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:52:31.993769 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:31.993718    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:52:32.508072 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:32.508046    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:32.510042 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:32.510012    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-86910b\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:32.601354 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:32.601241    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nzvt\" (UniqueName: \"kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt\") pod \"872924e5-c0b9-4d01-b082-0880c9940e80\" (UID: \"872924e5-c0b9-4d01-b082-0880c9940e80\") "
Sep 15 07:52:32.603518 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:32.603468    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt" (OuterVolumeSpecName: "kube-api-access-9nzvt") pod "872924e5-c0b9-4d01-b082-0880c9940e80" (UID: "872924e5-c0b9-4d01-b082-0880c9940e80"). InnerVolumeSpecName "kube-api-access-9nzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:52:32.702617 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:32.702578    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzvt\" (UniqueName: \"kubernetes.io/projected/872924e5-c0b9-4d01-b082-0880c9940e80-kube-api-access-9nzvt\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:33.332888 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:33.332847    2574 generic.go:356] "Generic (PLEG): container finished" podID="872924e5-c0b9-4d01-b082-0880c9940e80" containerID="06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72" exitCode=137
Sep 15 07:52:33.333365 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:33.332896    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:33.333365 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:33.332947    2574 scope.go:117] "RemoveContainer" containerID="06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72"
Sep 15 07:52:33.335109 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:33.335086    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-86910b\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:33.342766 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:33.342748    2574 scope.go:117] "RemoveContainer" containerID="06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72"
Sep 15 07:52:33.343080 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:33.343058    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72\": container with ID starting with 06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72 not found: ID does not exist" containerID="06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72"
Sep 15 07:52:33.343176 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:33.343089    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72"} err="failed to get container status \"06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72\": rpc error: code = NotFound desc = could not find container \"06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72\": container with ID starting with 06c3c747ae7e9627675d77414ab8e262bec6f1d1d054709c09bbf50550bfed72 not found: ID does not exist"
Sep 15 07:52:33.344176 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:33.344089    2574 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7660-86910b\" is forbidden: User \"system:node:ip-10-0-141-227.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-227.ec2.internal' and this object" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" pod="opendatahub/test-curl-7660-86910b"
Sep 15 07:52:34.378645 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:34.378611    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" path="/var/lib/kubelet/pods/872924e5-c0b9-4d01-b082-0880c9940e80/volumes"
Sep 15 07:52:35.203773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:35.203724    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:52:45.922230 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:45.922176    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" podUID="45c2d103-0376-42e2-a044-bae946bbf232" containerName="main" containerID="cri-o://032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6" gracePeriod=45
Sep 15 07:52:46.186492 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.186420    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:52:46.329251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329210    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329251 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329254    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329279    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329301    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nfzc\" (UniqueName: \"kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329520 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329484    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329547    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm\") pod \"45c2d103-0376-42e2-a044-bae946bbf232\" (UID: \"45c2d103-0376-42e2-a044-bae946bbf232\") "
Sep 15 07:52:46.329687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329553    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home" (OuterVolumeSpecName: "home") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:46.329687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329570    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache" (OuterVolumeSpecName: "model-cache") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:46.329844 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329726    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:46.329901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329859    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.329901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329872    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.329901 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.329880    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.331597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.331564    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm" (OuterVolumeSpecName: "dshm") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:46.331715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.331625    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc" (OuterVolumeSpecName: "kube-api-access-6nfzc") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "kube-api-access-6nfzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:52:46.331715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.331626    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45c2d103-0376-42e2-a044-bae946bbf232" (UID: "45c2d103-0376-42e2-a044-bae946bbf232"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:52:46.418442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.418404    2574 generic.go:356] "Generic (PLEG): container finished" podID="45c2d103-0376-42e2-a044-bae946bbf232" containerID="032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6" exitCode=0
Sep 15 07:52:46.418442 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.418442    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"
Sep 15 07:52:46.418644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.418490    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" event={"ID":"45c2d103-0376-42e2-a044-bae946bbf232","Type":"ContainerDied","Data":"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"}
Sep 15 07:52:46.418644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.418538    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477" event={"ID":"45c2d103-0376-42e2-a044-bae946bbf232","Type":"ContainerDied","Data":"6401ae08afdd226885c90e63939c9508d80936f2ec51131196012a3d7e2ab670"}
Sep 15 07:52:46.418644 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.418555    2574 scope.go:117] "RemoveContainer" containerID="032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"
Sep 15 07:52:46.427867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.427840    2574 scope.go:117] "RemoveContainer" containerID="032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"
Sep 15 07:52:46.428207 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:46.428165    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6\": container with ID starting with 032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6 not found: ID does not exist" containerID="032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"
Sep 15 07:52:46.428328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.428212    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6"} err="failed to get container status \"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6\": rpc error: code = NotFound desc = could not find container \"032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6\": container with ID starting with 032a9c83a69c97ac1670b02255c8851b522ea7442c3e746370edcdcda67726b6 not found: ID does not exist"
Sep 15 07:52:46.430653 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.430632    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45c2d103-0376-42e2-a044-bae946bbf232-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.430719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.430655    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfzc\" (UniqueName: \"kubernetes.io/projected/45c2d103-0376-42e2-a044-bae946bbf232-kube-api-access-6nfzc\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.430719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.430665    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45c2d103-0376-42e2-a044-bae946bbf232-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:46.435098 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.434864    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:52:46.437282 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:46.437227    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-04464b/sub-test-model-04464b-kserve-68bf8c68d7-l9477"]
Sep 15 07:52:47.014240 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.014178    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" podUID="07a64b16-4380-4768-a935-b5272005e998" containerName="main" containerID="cri-o://ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910" gracePeriod=45
Sep 15 07:52:47.261931 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.261899    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:52:47.339084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.338989    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339033    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339084 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339074    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbln\" (UniqueName: \"kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339139    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339168    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339251    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm\") pod \"07a64b16-4380-4768-a935-b5272005e998\" (UID: \"07a64b16-4380-4768-a935-b5272005e998\") "
Sep 15 07:52:47.339593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339420    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache" (OuterVolumeSpecName: "model-cache") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:47.339593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339432    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home" (OuterVolumeSpecName: "home") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:47.339593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339507    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:47.339593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339528    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.339593 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.339546    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.341570 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.341523    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm" (OuterVolumeSpecName: "dshm") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:52:47.341706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.341597    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:52:47.341706 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.341611    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln" (OuterVolumeSpecName: "kube-api-access-6nbln") pod "07a64b16-4380-4768-a935-b5272005e998" (UID: "07a64b16-4380-4768-a935-b5272005e998"). InnerVolumeSpecName "kube-api-access-6nbln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:52:47.429603 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.429566    2574 generic.go:356] "Generic (PLEG): container finished" podID="07a64b16-4380-4768-a935-b5272005e998" containerID="ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910" exitCode=0
Sep 15 07:52:47.429775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.429616    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"
Sep 15 07:52:47.429775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.429654    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" event={"ID":"07a64b16-4380-4768-a935-b5272005e998","Type":"ContainerDied","Data":"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"}
Sep 15 07:52:47.429775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.429700    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq" event={"ID":"07a64b16-4380-4768-a935-b5272005e998","Type":"ContainerDied","Data":"723e1f9f10653335433bf05fb034dd77c05c018020e77fbca84592df46c6ba3b"}
Sep 15 07:52:47.429775 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.429720    2574 scope.go:117] "RemoveContainer" containerID="ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"
Sep 15 07:52:47.439661 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.439634    2574 scope.go:117] "RemoveContainer" containerID="ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"
Sep 15 07:52:47.440001 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:52:47.439976    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910\": container with ID starting with ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910 not found: ID does not exist" containerID="ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"
Sep 15 07:52:47.440046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.440011    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910"} err="failed to get container status \"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910\": rpc error: code = NotFound desc = could not find container \"ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910\": container with ID starting with ac3dee2e61e5d82d958a494fdc0ee41503651914c793470c40f09129737c1910 not found: ID does not exist"
Sep 15 07:52:47.440091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.440043    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.440091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.440058    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07a64b16-4380-4768-a935-b5272005e998-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.440091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.440068    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbln\" (UniqueName: \"kubernetes.io/projected/07a64b16-4380-4768-a935-b5272005e998-kube-api-access-6nbln\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.440091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.440077    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07a64b16-4380-4768-a935-b5272005e998-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:52:47.452752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.452712    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:52:47.456244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.456209    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/sub-test-model-e3a482-kserve-5c867f845c-fcgpq"]
Sep 15 07:52:47.805532 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:47.805487    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:52:48.380052 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:48.380010    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07a64b16-4380-4768-a935-b5272005e998" path="/var/lib/kubelet/pods/07a64b16-4380-4768-a935-b5272005e998/volumes"
Sep 15 07:52:48.380497 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:52:48.380480    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c2d103-0376-42e2-a044-bae946bbf232" path="/var/lib/kubelet/pods/45c2d103-0376-42e2-a044-bae946bbf232/volumes"
Sep 15 07:53:20.810560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810513    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810900    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c2d103-0376-42e2-a044-bae946bbf232" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810912    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c2d103-0376-42e2-a044-bae946bbf232" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810921    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07a64b16-4380-4768-a935-b5272005e998" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810926    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="07a64b16-4380-4768-a935-b5272005e998" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810946    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" containerName="test-curl-7660-86910b"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.810952    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" containerName="test-curl-7660-86910b"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.811005    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c2d103-0376-42e2-a044-bae946bbf232" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.811013    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="07a64b16-4380-4768-a935-b5272005e998" containerName="main"
Sep 15 07:53:20.811065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.811021    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="872924e5-c0b9-4d01-b082-0880c9940e80" containerName="test-curl-7660-86910b"
Sep 15 07:53:20.814425 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.814399    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.816640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.816612    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"kube-root-ca.crt\""
Sep 15 07:53:20.817301 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.817276    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"openshift-service-ca.crt\""
Sep 15 07:53:20.817435 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.817299    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"default-dockercfg-jgj2b\""
Sep 15 07:53:20.817435 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.817344    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e3a482\"/\"rate-test-model-e3a482-kserve-self-signed-certs\""
Sep 15 07:53:20.823156 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.823099    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:53:20.951757 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951719    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.951925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951769    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.951925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951799    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.951925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951877    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.951925 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951920    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jx4z\" (UniqueName: \"kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:20.952066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:20.951991    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.052599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052552    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.052599 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052607    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.052870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052633    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.052870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.052870 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052803    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.053033 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.052845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jx4z\" (UniqueName: \"kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.053130 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.053086    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.053458 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.053433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.053553 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.053515    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.055308 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.055287    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.055432 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.055415    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.060379 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.060356    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jx4z\" (UniqueName: \"kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z\") pod \"rate-test-model-e3a482-kserve-6d77f5b495-vdjft\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") " pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.125212 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.125086    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:21.409778 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.409746    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:53:21.411612 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:53:21.411584    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cc62a88_7ea2_4b01_bf4a_400908f2bf39.slice/crio-8a1c237c3281c435e184cf1bc0b824f1de4f863173796c57dbf49ca8a15777de WatchSource:0}: Error finding container 8a1c237c3281c435e184cf1bc0b824f1de4f863173796c57dbf49ca8a15777de: Status 404 returned error can't find the container with id 8a1c237c3281c435e184cf1bc0b824f1de4f863173796c57dbf49ca8a15777de
Sep 15 07:53:21.626749 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.626704    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" event={"ID":"4cc62a88-7ea2-4b01-bf4a-400908f2bf39","Type":"ContainerStarted","Data":"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"}
Sep 15 07:53:21.626917 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.626760    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" event={"ID":"4cc62a88-7ea2-4b01-bf4a-400908f2bf39","Type":"ContainerStarted","Data":"8a1c237c3281c435e184cf1bc0b824f1de4f863173796c57dbf49ca8a15777de"}
Sep 15 07:53:21.645341 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:21.645281    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" podStartSLOduration=1.6452616070000001 podStartE2EDuration="1.645261607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:53:21.642368552 +0000 UTC m=+2089.736534804" watchObservedRunningTime="2026-09-15 07:53:21.645261607 +0000 UTC m=+2089.739427908"
Sep 15 07:53:25.017971 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:25.017930    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:53:31.126218 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:31.126171    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:31.126707 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:31.126242    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:31.138966 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:31.138928    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:31.698542 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:31.698506    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:53:45.710334 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:45.710292    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:53:55.605896 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:53:55.605855    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:54:02.923040 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:02.923004    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:54:03.607677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.607638    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:54:03.611919 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.611893    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.614299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.614273    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-b179ed-openshift-default-dockercfg-82xmn\""
Sep 15 07:54:03.623427 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.623394    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:54:03.658049 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658068    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658096    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658179    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658295 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd6xk\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658288    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658364    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.658481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.658441    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.759822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759774    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd6xk\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.759822 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759866    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759893    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759945    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.759984    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760020    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760101 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760090    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760379    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760398    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760817 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760789    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760955 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760813    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.760955 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.760858    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.762743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.762719    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.762863 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.762743    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.767566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.767535    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.767667 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.767565    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd6xk\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk\") pod \"e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") " pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:03.923043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:03.923006    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:04.201751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.201722    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:54:04.208150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.208095    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:04.208288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.208187    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:04.208288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.208222    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:04.868001 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.867960    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerStarted","Data":"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"}
Sep 15 07:54:04.868252 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.868023    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerStarted","Data":"2a9fa2942921f276a2045d05b014a19c33dffd19e74aeec9d186707e8e7ec74a"}
Sep 15 07:54:04.889020 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.888964    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podStartSLOduration=1.888946378 podStartE2EDuration="1.888946378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:04.888553903 +0000 UTC m=+2132.982720166" watchObservedRunningTime="2026-09-15 07:54:04.888946378 +0000 UTC m=+2132.983112638"
Sep 15 07:54:04.923345 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:04.923294    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:05.928410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:05.928373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:06.718711 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:06.718672    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:54:06.885740 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:06.885688    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:06.886872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:06.886840    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:08.329234 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.329195    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:54:08.332694 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.332674    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.335615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.335590    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b179ed-serving-cert\""
Sep 15 07:54:08.345436 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.345404    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:54:08.400440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.400396    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.400440 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.400444    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6q2r\" (UniqueName: \"kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.400670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.400527    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.400670 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.400609    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.502129 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.502067    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.502353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.502167    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.502353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.502195    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6q2r\" (UniqueName: \"kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.502353 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.502226    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.502353 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:08.502325    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-serving-cert: secret "maas-api-e2e-ipp-a-b179ed-serving-cert" not found
Sep 15 07:54:08.502590 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:08.502407    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls podName:fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4 nodeName:}" failed. No retries permitted until 2026-09-15 07:54:09.002385609 +0000 UTC m=+2137.096551852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls") pod "maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4") : secret "maas-api-e2e-ipp-a-b179ed-serving-cert" not found
Sep 15 07:54:08.502929 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.502910    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.504690 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.504665    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:08.511896 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:08.511867    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6q2r\" (UniqueName: \"kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:09.007371 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:09.007325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:09.007599 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:09.007529    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-serving-cert: secret "maas-api-e2e-ipp-a-b179ed-serving-cert" not found
Sep 15 07:54:09.007662 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:09.007623    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls podName:fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4 nodeName:}" failed. No retries permitted until 2026-09-15 07:54:10.007607178 +0000 UTC m=+2138.101773419 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls") pod "maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4") : secret "maas-api-e2e-ipp-a-b179ed-serving-cert" not found
Sep 15 07:54:10.016582 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.016533    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:10.019210 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.019180    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:10.145341 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.145299    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:10.423053 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.423027    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:54:10.424562 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:54:10.424529    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9af7fc_0d54_4b51_a3c9_31f4838cd8f4.slice/crio-d8a6c2b00863434fe3febad0787dc7e8405c5076b1cfa528646e255527502006 WatchSource:0}: Error finding container d8a6c2b00863434fe3febad0787dc7e8405c5076b1cfa528646e255527502006: Status 404 returned error can't find the container with id d8a6c2b00863434fe3febad0787dc7e8405c5076b1cfa528646e255527502006
Sep 15 07:54:10.426257 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.426231    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:54:10.917451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:10.917363    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" event={"ID":"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4","Type":"ContainerStarted","Data":"d8a6c2b00863434fe3febad0787dc7e8405c5076b1cfa528646e255527502006"}
Sep 15 07:54:11.925886 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:11.925838    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" event={"ID":"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4","Type":"ContainerStarted","Data":"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"}
Sep 15 07:54:11.926378 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:11.926054    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:11.942528 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:11.942472    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" podStartSLOduration=3.004789079 podStartE2EDuration="3.942455832s" totalImagesPullingTime="937.666753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:11.941176884 +0000 UTC m=+2140.035343146" watchObservedRunningTime="2026-09-15 07:54:11.942455832 +0000 UTC m=+2140.036622088"
Sep 15 07:54:17.937335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:17.937305    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:54:17.943639 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:17.943602    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" podUID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" containerName="main" containerID="cri-o://f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811" gracePeriod=45
Sep 15 07:54:18.245377 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.245345    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:54:18.401928 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.401887    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.401928 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.401927    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.402235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.401970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.402235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402022    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.402235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402085    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jx4z\" (UniqueName: \"kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.402235 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402134    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs\") pod \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\" (UID: \"4cc62a88-7ea2-4b01-bf4a-400908f2bf39\") "
Sep 15 07:54:18.402421 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402230    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:18.402421 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402256    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:18.402421 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402345    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home" (OuterVolumeSpecName: "home") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:18.402564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402514    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.402564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402529    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.402564 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.402537    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.404315 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.404285    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z" (OuterVolumeSpecName: "kube-api-access-5jx4z") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "kube-api-access-5jx4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:54:18.404431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.404348    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm" (OuterVolumeSpecName: "dshm") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:54:18.404431 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.404399    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cc62a88-7ea2-4b01-bf4a-400908f2bf39" (UID: "4cc62a88-7ea2-4b01-bf4a-400908f2bf39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:54:18.504150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.504014    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jx4z\" (UniqueName: \"kubernetes.io/projected/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-kube-api-access-5jx4z\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.504150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.504056    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.504150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.504067    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc62a88-7ea2-4b01-bf4a-400908f2bf39-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:54:18.999134 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.999077    2574 generic.go:356] "Generic (PLEG): container finished" podID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" containerID="f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811" exitCode=0
Sep 15 07:54:18.999579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.999145    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"
Sep 15 07:54:18.999579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.999163    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" event={"ID":"4cc62a88-7ea2-4b01-bf4a-400908f2bf39","Type":"ContainerDied","Data":"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"}
Sep 15 07:54:18.999579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.999216    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft" event={"ID":"4cc62a88-7ea2-4b01-bf4a-400908f2bf39","Type":"ContainerDied","Data":"8a1c237c3281c435e184cf1bc0b824f1de4f863173796c57dbf49ca8a15777de"}
Sep 15 07:54:18.999579 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:18.999233    2574 scope.go:117] "RemoveContainer" containerID="f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"
Sep 15 07:54:19.009467 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:19.009444    2574 scope.go:117] "RemoveContainer" containerID="f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"
Sep 15 07:54:19.009788 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:19.009768    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811\": container with ID starting with f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811 not found: ID does not exist" containerID="f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"
Sep 15 07:54:19.009897 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:19.009798    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811"} err="failed to get container status \"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811\": rpc error: code = NotFound desc = could not find container \"f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811\": container with ID starting with f2d674ec48721f753b842458530d36723ae83513317a9dba1ce2657afd6fb811 not found: ID does not exist"
Sep 15 07:54:19.020492 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:19.020459    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:54:19.024674 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:19.024636    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e3a482/rate-test-model-e3a482-kserve-6d77f5b495-vdjft"]
Sep 15 07:54:19.701323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:19.701282    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:54:20.380135 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:20.380089    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" path="/var/lib/kubelet/pods/4cc62a88-7ea2-4b01-bf4a-400908f2bf39/volumes"
Sep 15 07:54:22.295818 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:22.295436    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": read tcp 10.133.0.2:46748->10.133.0.90:15021: read: connection reset by peer" start-of-body=
Sep 15 07:54:22.295818 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:22.295531    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": read tcp 10.133.0.2:46748->10.133.0.90:15021: read: connection reset by peer"
Sep 15 07:54:23.025164 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.025109    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerID="cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2" exitCode=0
Sep 15 07:54:23.025352 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.025184    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerDied","Data":"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"}
Sep 15 07:54:23.025676 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.025661    2574 scope.go:117] "RemoveContainer" containerID="cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"
Sep 15 07:54:23.026851 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.026822    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:23.026914 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.026893    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:23.026951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:23.026925    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:24.032642 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:24.032600    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerStarted","Data":"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"}
Sep 15 07:54:24.924194 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:24.924153    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:24.924475 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:24.924450    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 15 07:54:24.924528 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:24.924507    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 15 07:54:25.924471 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:25.924426    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 15 07:54:25.924922 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:25.924490    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 15 07:54:26.923655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:26.923608    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 15 07:54:26.923857 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:26.923671    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 15 07:54:27.924465 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:27.924423    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 15 07:54:27.924881 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:27.924488    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 15 07:54:28.923566 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:28.923524    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Sep 15 07:54:28.923779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:28.923603    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Sep 15 07:54:29.927711 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:29.927668    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:29.928237 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:29.928066    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:29.928868 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:29.928846    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:54:32.566648 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.566607    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:54:32.567206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.567185    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" containerName="main"
Sep 15 07:54:32.567299 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.567207    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" containerName="main"
Sep 15 07:54:32.567369 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.567305    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc62a88-7ea2-4b01-bf4a-400908f2bf39" containerName="main"
Sep 15 07:54:32.571065 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.571037    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.573225 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.573197    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5d58bd-openshift-default-dockercfg-dgrgd\""
Sep 15 07:54:32.584072 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.584040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:54:32.639389 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639409    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639448    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639485    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639511    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639540    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66gm6\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639573    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639597    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.639632 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.639611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.740741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.740741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740739    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740783    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66gm6\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740822    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740852    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740875    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741007 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.740997    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741351 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741022    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741351 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741351 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741304    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741494 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741475    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741555 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741533    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.741741 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.741723    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.743535 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.743453    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.743849 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.743827    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.750678 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.750641    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.750826 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.750753    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66gm6\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6\") pod \"e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") " pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:32.882386 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:32.882290    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:33.200830 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:33.200781    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:54:33.201638 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:54:33.201608    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156cae51_7ea5_4f2a_b58a_715bd56e93f2.slice/crio-5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809 WatchSource:0}: Error finding container 5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809: Status 404 returned error can't find the container with id 5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809
Sep 15 07:54:33.204093 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:33.204050    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:33.204241 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:33.204153    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:33.204241 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:33.204205    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:34.097669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:34.097628    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerStarted","Data":"ea048e4e12b452e50de61a6be3928c419c21f4bbdda9dce941211a1131f6b7d8"}
Sep 15 07:54:34.098311 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:34.097685    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerStarted","Data":"5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809"}
Sep 15 07:54:34.116622 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:34.116561    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podStartSLOduration=2.116540723 podStartE2EDuration="2.116540723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:34.115002352 +0000 UTC m=+2162.209168614" watchObservedRunningTime="2026-09-15 07:54:34.116540723 +0000 UTC m=+2162.210707052"
Sep 15 07:54:34.882867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:34.882823    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:34.888602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:34.888567    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:35.102863 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:35.102822    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:35.103751 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:35.103732    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:38.818884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.818844    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:54:38.823609 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.823582    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:38.825602 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.825572    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5d58bd-serving-cert\""
Sep 15 07:54:38.830330 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.830297    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:54:38.914227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.914178    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:38.914417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.914269    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:38.914417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.914351    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8rg2\" (UniqueName: \"kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:38.914417 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:38.914401    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.015046 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.015007    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.015284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.015052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.015284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.015111    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.015284 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:39.015192    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-serving-cert: secret "maas-api-e2e-ipp-b-5d58bd-serving-cert" not found
Sep 15 07:54:39.015284 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.015258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8rg2\" (UniqueName: \"kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.015284 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:39.015280    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls podName:169902f1-40e3-4c9f-bbab-67709f0583fb nodeName:}" failed. No retries permitted until 2026-09-15 07:54:39.51525835 +0000 UTC m=+2167.609424604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls") pod "maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb") : secret "maas-api-e2e-ipp-b-5d58bd-serving-cert" not found
Sep 15 07:54:39.015779 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.015756    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.017760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.017741    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.026079 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.026049    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8rg2\" (UniqueName: \"kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.520858 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:39.520821    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:39.521046 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:39.520997    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-serving-cert: secret "maas-api-e2e-ipp-b-5d58bd-serving-cert" not found
Sep 15 07:54:39.521092 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:54:39.521070    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls podName:169902f1-40e3-4c9f-bbab-67709f0583fb nodeName:}" failed. No retries permitted until 2026-09-15 07:54:40.521054949 +0000 UTC m=+2168.615221194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls") pod "maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb") : secret "maas-api-e2e-ipp-b-5d58bd-serving-cert" not found
Sep 15 07:54:40.529812 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:40.529777    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:40.532297 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:40.532268    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:40.638613 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:40.638569    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:40.921289 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:40.921256    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:54:40.922964 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:54:40.922932    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169902f1_40e3_4c9f_bbab_67709f0583fb.slice/crio-b6658dd6adcbd9b4b98d5e2530df669e035dd0ee03a46312aa8ecc014e4f49b2 WatchSource:0}: Error finding container b6658dd6adcbd9b4b98d5e2530df669e035dd0ee03a46312aa8ecc014e4f49b2: Status 404 returned error can't find the container with id b6658dd6adcbd9b4b98d5e2530df669e035dd0ee03a46312aa8ecc014e4f49b2
Sep 15 07:54:41.137885 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:41.137840    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" event={"ID":"169902f1-40e3-4c9f-bbab-67709f0583fb","Type":"ContainerStarted","Data":"b6658dd6adcbd9b4b98d5e2530df669e035dd0ee03a46312aa8ecc014e4f49b2"}
Sep 15 07:54:42.144874 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:42.144770    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" event={"ID":"169902f1-40e3-4c9f-bbab-67709f0583fb","Type":"ContainerStarted","Data":"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"}
Sep 15 07:54:42.144874 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:42.144846    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:42.164224 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:42.164159    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" podStartSLOduration=3.24807709 podStartE2EDuration="4.164138193s" totalImagesPullingTime="916.061103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:54:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:54:42.161435715 +0000 UTC m=+2170.255601980" watchObservedRunningTime="2026-09-15 07:54:42.164138193 +0000 UTC m=+2170.258304452"
Sep 15 07:54:44.811186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:44.811149    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:54:48.160489 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:48.160454    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:54:55.223806 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.223705    2574 generic.go:356] "Generic (PLEG): container finished" podID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerID="ea048e4e12b452e50de61a6be3928c419c21f4bbdda9dce941211a1131f6b7d8" exitCode=0
Sep 15 07:54:55.223806 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.223757    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerDied","Data":"ea048e4e12b452e50de61a6be3928c419c21f4bbdda9dce941211a1131f6b7d8"}
Sep 15 07:54:55.224285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.224227    2574 scope.go:117] "RemoveContainer" containerID="ea048e4e12b452e50de61a6be3928c419c21f4bbdda9dce941211a1131f6b7d8"
Sep 15 07:54:55.225472 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.225438    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:55.225549 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.225515    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:55.225591 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:55.225557    2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 15 07:54:56.231062 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:56.231024    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerStarted","Data":"ff9918ec86b500ea6c113e2ac06e8db2abec39f9b4b995f6801a8e1cfed74bee"}
Sep 15 07:54:56.882969 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:56.882921    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:54:56.883280 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:56.883254    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 07:54:56.883447 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:56.883318    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 07:54:57.883285 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:57.883234    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 07:54:57.883780 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:57.883301    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 07:54:58.882744 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:58.882699    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 07:54:58.882951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:58.882760    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 07:54:59.883021 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:59.882981    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 07:54:59.883458 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:54:59.883042    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 07:55:00.883596 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:00.883558    2574 patch_prober.go:28] interesting pod/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body=
Sep 15 07:55:00.884094 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:00.883618    2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused"
Sep 15 07:55:01.887237 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:01.887205    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:55:01.887628 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:01.887503    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:55:01.888191 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:01.888169    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:55:19.806894 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.806851    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:55:19.811701 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.811671    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.814723 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.814693    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b179ed\"/\"default-dockercfg-mwmdx\""
Sep 15 07:55:19.814905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.814733    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b179ed\"/\"kube-root-ca.crt\""
Sep 15 07:55:19.814905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.814694    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-b179ed\"/\"ipp-route-b179ed-kserve-self-signed-certs\""
Sep 15 07:55:19.814905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.814699    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-b179ed\"/\"openshift-service-ca.crt\""
Sep 15 07:55:19.819908 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.819882    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:55:19.908770 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.908726    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.908992 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.908868    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.908992 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.908917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.908992 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.908940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.908992 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.908957    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vmdp\" (UniqueName: \"kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:19.909288 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:19.909029    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010002 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.009952    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010044    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010151    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010177    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vmdp\" (UniqueName: \"kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010263 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010211    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010561 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010489    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010561 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010546    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.010669 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.010620    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.012481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.012462    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.012583 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.012567    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.017767 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.017734    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vmdp\" (UniqueName: \"kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp\") pod \"ipp-route-b179ed-kserve-77b97cf7f5-mfn2h\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") " pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.124770 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.124659    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:20.401509 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:20.401478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:55:20.403284 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:55:20.403258    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69e9f937_8105_45ed_ac90_241b51452bac.slice/crio-0cd0138e63015a98fce8159c03c7ea47907de52602967f2988c14b5cf063b8d2 WatchSource:0}: Error finding container 0cd0138e63015a98fce8159c03c7ea47907de52602967f2988c14b5cf063b8d2: Status 404 returned error can't find the container with id 0cd0138e63015a98fce8159c03c7ea47907de52602967f2988c14b5cf063b8d2
Sep 15 07:55:21.390936 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:21.390891    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" event={"ID":"69e9f937-8105-45ed-ac90-241b51452bac","Type":"ContainerStarted","Data":"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"}
Sep 15 07:55:21.390936 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:21.390941    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" event={"ID":"69e9f937-8105-45ed-ac90-241b51452bac","Type":"ContainerStarted","Data":"0cd0138e63015a98fce8159c03c7ea47907de52602967f2988c14b5cf063b8d2"}
Sep 15 07:55:21.413210 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:21.413151    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" podStartSLOduration=2.413134506 podStartE2EDuration="2.413134506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:55:21.410634582 +0000 UTC m=+2209.504800855" watchObservedRunningTime="2026-09-15 07:55:21.413134506 +0000 UTC m=+2209.507300788"
Sep 15 07:55:24.105872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:24.105830    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:55:30.125179 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:30.125108    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:30.125722 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:30.125203    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:30.138206 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:30.138173    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:30.455466 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:30.455432    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:55:37.609848 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:37.609802    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:55:52.011152 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.011100    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:55:52.014738 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.014712    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.017022 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.016989    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-0fd0de-serving-cert\""
Sep 15 07:55:52.023199 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.023035    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.023199 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.023188    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.023411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.023275    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r76n\" (UniqueName: \"kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.023411 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.023320    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.025324 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.025271    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:55:52.124006 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.123967    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r76n\" (UniqueName: \"kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.124336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.124031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.124336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.124093    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.124336 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.124178    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.124336 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:55:52.124311    2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-serving-cert: secret "maas-api-e2e-ipp-cleanup-0fd0de-serving-cert" not found
Sep 15 07:55:52.124572 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:55:52.124392    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls podName:335b69e1-83c0-48dc-932c-5bd1a9df8a37 nodeName:}" failed. No retries permitted until 2026-09-15 07:55:52.62436851 +0000 UTC m=+2240.718534751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" (UID: "335b69e1-83c0-48dc-932c-5bd1a9df8a37") : secret "maas-api-e2e-ipp-cleanup-0fd0de-serving-cert" not found
Sep 15 07:55:52.125048 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.125003    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.127926 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.127894    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.134433 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.134401    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r76n\" (UniqueName: \"kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.628622 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.628558    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.631253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.631219    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:52.929174 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:52.929099    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:53.209095 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:53.209064    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:55:53.210942 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:55:53.210906    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335b69e1_83c0_48dc_932c_5bd1a9df8a37.slice/crio-08eda069c0b988dbe6320bfad65d8df6bf7129d178919a1dac9b1efe8dfb64d1 WatchSource:0}: Error finding container 08eda069c0b988dbe6320bfad65d8df6bf7129d178919a1dac9b1efe8dfb64d1: Status 404 returned error can't find the container with id 08eda069c0b988dbe6320bfad65d8df6bf7129d178919a1dac9b1efe8dfb64d1
Sep 15 07:55:53.584307 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:53.584185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" event={"ID":"335b69e1-83c0-48dc-932c-5bd1a9df8a37","Type":"ContainerStarted","Data":"08eda069c0b988dbe6320bfad65d8df6bf7129d178919a1dac9b1efe8dfb64d1"}
Sep 15 07:55:56.603598 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:56.603553    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" event={"ID":"335b69e1-83c0-48dc-932c-5bd1a9df8a37","Type":"ContainerStarted","Data":"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"}
Sep 15 07:55:56.604066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:56.603645    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:55:56.621988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:56.621925    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" podStartSLOduration=3.131352709 podStartE2EDuration="5.621905843s" totalImagesPullingTime="2.490553134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:55:56.619875294 +0000 UTC m=+2244.714041568" watchObservedRunningTime="2026-09-15 07:55:56.621905843 +0000 UTC m=+2244.716072169"
Sep 15 07:55:57.011954 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:55:57.011905    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:56:02.615043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:02.615010    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:56:30.069615 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.069573    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:56:30.070244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.069828    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" podUID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" containerName="maas-api" containerID="cri-o://2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf" gracePeriod=30
Sep 15 07:56:30.346606 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.346571    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:56:30.506468 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.506426    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") pod \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") "
Sep 15 07:56:30.506687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.506497    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca\") pod \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") "
Sep 15 07:56:30.506687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.506530    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs\") pod \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") "
Sep 15 07:56:30.506687 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.506550    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r76n\" (UniqueName: \"kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n\") pod \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\" (UID: \"335b69e1-83c0-48dc-932c-5bd1a9df8a37\") "
Sep 15 07:56:30.506867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.506838    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "335b69e1-83c0-48dc-932c-5bd1a9df8a37" (UID: "335b69e1-83c0-48dc-932c-5bd1a9df8a37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:30.508872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.508843    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "335b69e1-83c0-48dc-932c-5bd1a9df8a37" (UID: "335b69e1-83c0-48dc-932c-5bd1a9df8a37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:56:30.509066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.508920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "335b69e1-83c0-48dc-932c-5bd1a9df8a37" (UID: "335b69e1-83c0-48dc-932c-5bd1a9df8a37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:56:30.509066 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.509037    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n" (OuterVolumeSpecName: "kube-api-access-5r76n") pod "335b69e1-83c0-48dc-932c-5bd1a9df8a37" (UID: "335b69e1-83c0-48dc-932c-5bd1a9df8a37"). InnerVolumeSpecName "kube-api-access-5r76n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:56:30.607692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.607591    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.607692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.607628    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/335b69e1-83c0-48dc-932c-5bd1a9df8a37-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.607692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.607639    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/335b69e1-83c0-48dc-932c-5bd1a9df8a37-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.607692 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.607649    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r76n\" (UniqueName: \"kubernetes.io/projected/335b69e1-83c0-48dc-932c-5bd1a9df8a37-kube-api-access-5r76n\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:30.806186 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.806148    2574 generic.go:356] "Generic (PLEG): container finished" podID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" containerID="2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf" exitCode=0
Sep 15 07:56:30.806364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.806201    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" event={"ID":"335b69e1-83c0-48dc-932c-5bd1a9df8a37","Type":"ContainerDied","Data":"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"}
Sep 15 07:56:30.806364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.806249    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"
Sep 15 07:56:30.806364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.806257    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8" event={"ID":"335b69e1-83c0-48dc-932c-5bd1a9df8a37","Type":"ContainerDied","Data":"08eda069c0b988dbe6320bfad65d8df6bf7129d178919a1dac9b1efe8dfb64d1"}
Sep 15 07:56:30.806364 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.806288    2574 scope.go:117] "RemoveContainer" containerID="2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"
Sep 15 07:56:30.817004 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.816981    2574 scope.go:117] "RemoveContainer" containerID="2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"
Sep 15 07:56:30.817365 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:56:30.817342    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf\": container with ID starting with 2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf not found: ID does not exist" containerID="2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"
Sep 15 07:56:30.817482 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.817374    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf"} err="failed to get container status \"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf\": rpc error: code = NotFound desc = could not find container \"2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf\": container with ID starting with 2815d2b8a0ec0dd6d411b7c79b24141609c10bab49aa542fb926af7f8fbfb3cf not found: ID does not exist"
Sep 15 07:56:30.828612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.828576    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:56:30.834693 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:30.834663    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-0fd0de-6ff5758c74-qh5s8"]
Sep 15 07:56:32.378791 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:32.378755    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" path="/var/lib/kubelet/pods/335b69e1-83c0-48dc-932c-5bd1a9df8a37/volumes"
Sep 15 07:56:34.807619 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:34.807572    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:56:53.714790 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:53.714747    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:56:55.991244 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:55.991203    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:56:55.991746 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:55.991486    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" podUID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" containerName="maas-api" containerID="cri-o://ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f" gracePeriod=30
Sep 15 07:56:56.262328 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.262295    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:56:56.354967 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.354932    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6q2r\" (UniqueName: \"kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r\") pod \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") "
Sep 15 07:56:56.355196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.354979    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") pod \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") "
Sep 15 07:56:56.355196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.355013    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca\") pod \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") "
Sep 15 07:56:56.355196 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.355080    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs\") pod \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\" (UID: \"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4\") "
Sep 15 07:56:56.356063 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.355943    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:56:56.358081 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.358043    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:56:56.358253 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.358223    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:56:56.358481 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.358451    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r" (OuterVolumeSpecName: "kube-api-access-q6q2r") pod "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" (UID: "fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4"). InnerVolumeSpecName "kube-api-access-q6q2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:56:56.456988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.456933    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6q2r\" (UniqueName: \"kubernetes.io/projected/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-kube-api-access-q6q2r\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.456988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.456984    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.456988 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.456999    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.457348 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.457014    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:56:56.963031 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.962990    2574 generic.go:356] "Generic (PLEG): container finished" podID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" containerID="ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f" exitCode=0
Sep 15 07:56:56.963242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.963042    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" event={"ID":"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4","Type":"ContainerDied","Data":"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"}
Sep 15 07:56:56.963242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.963058    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"
Sep 15 07:56:56.963242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.963148    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r" event={"ID":"fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4","Type":"ContainerDied","Data":"d8a6c2b00863434fe3febad0787dc7e8405c5076b1cfa528646e255527502006"}
Sep 15 07:56:56.963242 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.963180    2574 scope.go:117] "RemoveContainer" containerID="ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"
Sep 15 07:56:56.973550 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.973523    2574 scope.go:117] "RemoveContainer" containerID="ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"
Sep 15 07:56:56.973883 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:56:56.973860    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f\": container with ID starting with ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f not found: ID does not exist" containerID="ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"
Sep 15 07:56:56.973945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.973892    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f"} err="failed to get container status \"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f\": rpc error: code = NotFound desc = could not find container \"ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f\": container with ID starting with ff8e64bdd372e3ab232e9421b4128ff82b995cc20f0f9b8e5858b452d03d470f not found: ID does not exist"
Sep 15 07:56:56.978918 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.978882    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:56:56.982713 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:56.982685    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b179ed-866cbdc474-xz69r"]
Sep 15 07:56:58.379446 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:56:58.379410    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" path="/var/lib/kubelet/pods/fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4/volumes"
Sep 15 07:57:01.561451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:01.561411    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:57:07.308451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:07.308419    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:57:16.580732 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.580649    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" podUID="69e9f937-8105-45ed-ac90-241b51452bac" containerName="main" containerID="cri-o://a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b" gracePeriod=45
Sep 15 07:57:16.828879 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.828852    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:57:16.955560 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955527    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955599    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955653    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955681    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955739 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955708    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vmdp\" (UniqueName: \"kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955968 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955755    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs\") pod \"69e9f937-8105-45ed-ac90-241b51452bac\" (UID: \"69e9f937-8105-45ed-ac90-241b51452bac\") "
Sep 15 07:57:16.955968 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955880    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:16.955968 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home" (OuterVolumeSpecName: "home") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:16.956159 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.955992    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache" (OuterVolumeSpecName: "model-cache") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:16.956223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.956191    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:16.956223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.956208    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:16.956223 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.956217    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:16.957945 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.957920    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm" (OuterVolumeSpecName: "dshm") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:16.958061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.957918    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:57:16.958061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:16.957957    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp" (OuterVolumeSpecName: "kube-api-access-7vmdp") pod "69e9f937-8105-45ed-ac90-241b51452bac" (UID: "69e9f937-8105-45ed-ac90-241b51452bac"). InnerVolumeSpecName "kube-api-access-7vmdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:17.057089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.057056    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vmdp\" (UniqueName: \"kubernetes.io/projected/69e9f937-8105-45ed-ac90-241b51452bac-kube-api-access-7vmdp\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:17.057089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.057085    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e9f937-8105-45ed-ac90-241b51452bac-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:17.057089 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.057094    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69e9f937-8105-45ed-ac90-241b51452bac-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:17.076034 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.075999    2574 generic.go:356] "Generic (PLEG): container finished" podID="69e9f937-8105-45ed-ac90-241b51452bac" containerID="a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b" exitCode=0
Sep 15 07:57:17.076250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.076044    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"
Sep 15 07:57:17.076250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.076056    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" event={"ID":"69e9f937-8105-45ed-ac90-241b51452bac","Type":"ContainerDied","Data":"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"}
Sep 15 07:57:17.076250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.076097    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h" event={"ID":"69e9f937-8105-45ed-ac90-241b51452bac","Type":"ContainerDied","Data":"0cd0138e63015a98fce8159c03c7ea47907de52602967f2988c14b5cf063b8d2"}
Sep 15 07:57:17.076250 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.076141    2574 scope.go:117] "RemoveContainer" containerID="a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"
Sep 15 07:57:17.085760 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.085742    2574 scope.go:117] "RemoveContainer" containerID="a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"
Sep 15 07:57:17.086030 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:57:17.086009    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b\": container with ID starting with a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b not found: ID does not exist" containerID="a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"
Sep 15 07:57:17.086082 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.086038    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b"} err="failed to get container status \"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b\": rpc error: code = NotFound desc = could not find container \"a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b\": container with ID starting with a063322fa77471b9a97381030f240fd0fba9660e48e2d3ae663bbe1e3631a28b not found: ID does not exist"
Sep 15 07:57:17.097664 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.097642    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:57:17.103335 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:17.103313    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-b179ed/ipp-route-b179ed-kserve-77b97cf7f5-mfn2h"]
Sep 15 07:57:18.379410 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:18.379380    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e9f937-8105-45ed-ac90-241b51452bac" path="/var/lib/kubelet/pods/69e9f937-8105-45ed-ac90-241b51452bac/volumes"
Sep 15 07:57:44.287750 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:44.287715    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:57:44.288261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:44.287986    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" containerID="cri-o://0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109" gracePeriod=30
Sep 15 07:57:44.930538 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:44.930504    2574 patch_prober.go:28] interesting pod/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 07:57:44.930711 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:44.930575    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 07:57:49.338396 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:57:49.338369    2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d40bbc7_19c9_478f_bf92_99ab2afd1a30.slice/crio-conmon-0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109.scope\": RecentStats: unable to find data in memory cache]"
Sep 15 07:57:49.532861 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.532833    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:57:49.646408 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646339    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646408 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646374    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646408 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646435    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646451    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646476    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646495    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd6xk\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646553    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646595    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token\") pod \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\" (UID: \"7d40bbc7-19c9-478f-bf92-99ab2afd1a30\") "
Sep 15 07:57:49.646652 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646616    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:49.647015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646800    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data" (OuterVolumeSpecName: "istio-data") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:49.647015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:49.647015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646854    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.647015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646976    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:49.647015 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.646910    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:57:49.648772 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.648750    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk" (OuterVolumeSpecName: "kube-api-access-vd6xk") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "kube-api-access-vd6xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:49.648860 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.648774    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:57:49.648956 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.648940    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token" (OuterVolumeSpecName: "istio-token") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:49.648995 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.648945    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7d40bbc7-19c9-478f-bf92-99ab2afd1a30" (UID: "7d40bbc7-19c9-478f-bf92-99ab2afd1a30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:57:49.748512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748487    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748512 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748511    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748524    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748533    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748542    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748550    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd6xk\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-kube-api-access-vd6xk\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748558    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:49.748668 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:49.748566    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d40bbc7-19c9-478f-bf92-99ab2afd1a30-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:50.251009 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.250977    2574 generic.go:356] "Generic (PLEG): container finished" podID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerID="0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109" exitCode=0
Sep 15 07:57:50.251180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.251028    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerDied","Data":"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"}
Sep 15 07:57:50.251180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.251038    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"
Sep 15 07:57:50.251180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.251081    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t" event={"ID":"7d40bbc7-19c9-478f-bf92-99ab2afd1a30","Type":"ContainerDied","Data":"2a9fa2942921f276a2045d05b014a19c33dffd19e74aeec9d186707e8e7ec74a"}
Sep 15 07:57:50.251180 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.251134    2574 scope.go:117] "RemoveContainer" containerID="0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"
Sep 15 07:57:50.260896 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.260879    2574 scope.go:117] "RemoveContainer" containerID="cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"
Sep 15 07:57:50.268903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.268886    2574 scope.go:117] "RemoveContainer" containerID="0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"
Sep 15 07:57:50.269161 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:57:50.269143    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109\": container with ID starting with 0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109 not found: ID does not exist" containerID="0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"
Sep 15 07:57:50.269226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.269170    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109"} err="failed to get container status \"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109\": rpc error: code = NotFound desc = could not find container \"0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109\": container with ID starting with 0012bfcd85cbaddb67ac8628977856cc75e112b38f7378f3b22b397dca3df109 not found: ID does not exist"
Sep 15 07:57:50.269226 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.269189    2574 scope.go:117] "RemoveContainer" containerID="cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"
Sep 15 07:57:50.269422 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:57:50.269408    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2\": container with ID starting with cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2 not found: ID does not exist" containerID="cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"
Sep 15 07:57:50.269463 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.269425    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2"} err="failed to get container status \"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2\": rpc error: code = NotFound desc = could not find container \"cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2\": container with ID starting with cebf446c41e272992ff0dc47ab2de9bffede8350e00a0b721634e89c223fead2 not found: ID does not exist"
Sep 15 07:57:50.272464 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.272443    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:57:50.279614 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.279593    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-b179ed-openshift-default-85fcb5565d-v4q6t"]
Sep 15 07:57:50.378150 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:50.378110    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" path="/var/lib/kubelet/pods/7d40bbc7-19c9-478f-bf92-99ab2afd1a30/volumes"
Sep 15 07:57:55.391627 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.391560    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:57:55.393329 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.393281    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" podUID="169902f1-40e3-4c9f-bbab-67709f0583fb" containerName="maas-api" containerID="cri-o://0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3" gracePeriod=30
Sep 15 07:57:55.641743 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.641684    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:57:55.800490 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.800458    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") pod \"169902f1-40e3-4c9f-bbab-67709f0583fb\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") "
Sep 15 07:57:55.800677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.800512    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8rg2\" (UniqueName: \"kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2\") pod \"169902f1-40e3-4c9f-bbab-67709f0583fb\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") "
Sep 15 07:57:55.800677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.800568    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca\") pod \"169902f1-40e3-4c9f-bbab-67709f0583fb\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") "
Sep 15 07:57:55.800677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.800620    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs\") pod \"169902f1-40e3-4c9f-bbab-67709f0583fb\" (UID: \"169902f1-40e3-4c9f-bbab-67709f0583fb\") "
Sep 15 07:57:55.800963 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.800939    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "169902f1-40e3-4c9f-bbab-67709f0583fb" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:57:55.802909 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.802875    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "169902f1-40e3-4c9f-bbab-67709f0583fb" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:57:55.803214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.803185    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2" (OuterVolumeSpecName: "kube-api-access-s8rg2") pod "169902f1-40e3-4c9f-bbab-67709f0583fb" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb"). InnerVolumeSpecName "kube-api-access-s8rg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:57:55.803214 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.803203    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "169902f1-40e3-4c9f-bbab-67709f0583fb" (UID: "169902f1-40e3-4c9f-bbab-67709f0583fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:57:55.902043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.901969    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8rg2\" (UniqueName: \"kubernetes.io/projected/169902f1-40e3-4c9f-bbab-67709f0583fb-kube-api-access-s8rg2\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:55.902043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.901996    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/169902f1-40e3-4c9f-bbab-67709f0583fb-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:55.902043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.902007    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:55.902043 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:55.902015    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/169902f1-40e3-4c9f-bbab-67709f0583fb-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:57:56.288867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.288833    2574 generic.go:356] "Generic (PLEG): container finished" podID="169902f1-40e3-4c9f-bbab-67709f0583fb" containerID="0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3" exitCode=0
Sep 15 07:57:56.289038 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.288875    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"
Sep 15 07:57:56.289038 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.288884    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" event={"ID":"169902f1-40e3-4c9f-bbab-67709f0583fb","Type":"ContainerDied","Data":"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"}
Sep 15 07:57:56.289038 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.288925    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv" event={"ID":"169902f1-40e3-4c9f-bbab-67709f0583fb","Type":"ContainerDied","Data":"b6658dd6adcbd9b4b98d5e2530df669e035dd0ee03a46312aa8ecc014e4f49b2"}
Sep 15 07:57:56.289038 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.288945    2574 scope.go:117] "RemoveContainer" containerID="0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"
Sep 15 07:57:56.298883 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.298869    2574 scope.go:117] "RemoveContainer" containerID="0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"
Sep 15 07:57:56.299209 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:57:56.299184    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3\": container with ID starting with 0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3 not found: ID does not exist" containerID="0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"
Sep 15 07:57:56.299279 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.299222    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3"} err="failed to get container status \"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3\": rpc error: code = NotFound desc = could not find container \"0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3\": container with ID starting with 0ffc209e7f0fb355fd21301515e4a8c0df12c2e0d8de1723d8ef29617ada51c3 not found: ID does not exist"
Sep 15 07:57:56.318309 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.318284    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:57:56.323344 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.323319    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5d58bd-5cbb9db589-vwdtv"]
Sep 15 07:57:56.377923 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:56.377897    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169902f1-40e3-4c9f-bbab-67709f0583fb" path="/var/lib/kubelet/pods/169902f1-40e3-4c9f-bbab-67709f0583fb/volumes"
Sep 15 07:57:58.807488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:57:58.807453    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:58:02.107468 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:02.107436    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:58:02.107905 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:02.107686    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy" containerID="cri-o://ff9918ec86b500ea6c113e2ac06e8db2abec39f9b4b995f6801a8e1cfed74bee" gracePeriod=30
Sep 15 07:58:07.351261 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.351232    2574 generic.go:356] "Generic (PLEG): container finished" podID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerID="ff9918ec86b500ea6c113e2ac06e8db2abec39f9b4b995f6801a8e1cfed74bee" exitCode=0
Sep 15 07:58:07.351612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.351301    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerDied","Data":"ff9918ec86b500ea6c113e2ac06e8db2abec39f9b4b995f6801a8e1cfed74bee"}
Sep 15 07:58:07.351612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.351357    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz" event={"ID":"156cae51-7ea5-4f2a-b58a-715bd56e93f2","Type":"ContainerDied","Data":"5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809"}
Sep 15 07:58:07.351612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.351374    2574 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c874e88c76662870a04943bf291e51af57cd779bca2675a6023e7baa9ab0809"
Sep 15 07:58:07.351612 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.351387    2574 scope.go:117] "RemoveContainer" containerID="ea048e4e12b452e50de61a6be3928c419c21f4bbdda9dce941211a1131f6b7d8"
Sep 15 07:58:07.357626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.357605    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:58:07.399519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399461    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399519 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399506    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399526    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399635    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399694    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399725    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66gm6\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399777    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399805    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.399884 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399849    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo\") pod \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\" (UID: \"156cae51-7ea5-4f2a-b58a-715bd56e93f2\") "
Sep 15 07:58:07.400108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399880    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:07.400108 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.399898    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:58:07.400266 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400158    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data" (OuterVolumeSpecName: "istio-data") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:58:07.400402 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400374    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:58:07.400533 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400481    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:58:07.400903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400829    2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-credential-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.400903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400862    2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-data\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.400903 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.400877    2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istiod-ca-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.401854 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.401826    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token" (OuterVolumeSpecName: "istio-token") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:07.401996 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.401973    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 07:58:07.402087 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.402068    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6" (OuterVolumeSpecName: "kube-api-access-66gm6") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "kube-api-access-66gm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:07.402451 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.402427    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "156cae51-7ea5-4f2a-b58a-715bd56e93f2" (UID: "156cae51-7ea5-4f2a-b58a-715bd56e93f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 15 07:58:07.501715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501694    2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.501715 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501714    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66gm6\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-kube-api-access-66gm6\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.501832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501723    2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-workload-socket\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.501832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501732    2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-podinfo\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.501832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501739    2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-envoy\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:07.501832 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:07.501749    2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/156cae51-7ea5-4f2a-b58a-715bd56e93f2-istio-token\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:08.358655 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:08.358623    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"
Sep 15 07:58:08.389991 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:08.389954    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:58:08.393527 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:08.393502    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d58bd-openshift-default-95bc45568-rl5tz"]
Sep 15 07:58:10.379595 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:10.379561    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" path="/var/lib/kubelet/pods/156cae51-7ea5-4f2a-b58a-715bd56e93f2/volumes"
Sep 15 07:58:13.146934 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.146901    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:58:13.147430 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.147137    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" podUID="2f065738-983d-49d6-98fb-0906ed0492cd" containerName="maas-api" containerID="cri-o://7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5" gracePeriod=30
Sep 15 07:58:13.393867 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.393843    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:58:13.394710 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.394686    2574 generic.go:356] "Generic (PLEG): container finished" podID="2f065738-983d-49d6-98fb-0906ed0492cd" containerID="7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5" exitCode=0
Sep 15 07:58:13.394809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.394754    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" event={"ID":"2f065738-983d-49d6-98fb-0906ed0492cd","Type":"ContainerDied","Data":"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"}
Sep 15 07:58:13.394809 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.394802    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh" event={"ID":"2f065738-983d-49d6-98fb-0906ed0492cd","Type":"ContainerDied","Data":"9c4e76de402e3c41be2c1d4ee136809289f13a5bd97c70633dd370d9d0f71533"}
Sep 15 07:58:13.394880 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.394816    2574 scope.go:117] "RemoveContainer" containerID="7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"
Sep 15 07:58:13.403025 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.403004    2574 scope.go:117] "RemoveContainer" containerID="7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"
Sep 15 07:58:13.403342 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:58:13.403315    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5\": container with ID starting with 7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5 not found: ID does not exist" containerID="7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"
Sep 15 07:58:13.403392 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.403342    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5"} err="failed to get container status \"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5\": rpc error: code = NotFound desc = could not find container \"7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5\": container with ID starting with 7e648d419f52f062066fc734a50d5aacefd7452e2a232b22aaf83afaab5004f5 not found: ID does not exist"
Sep 15 07:58:13.447843 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.447817    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca\") pod \"2f065738-983d-49d6-98fb-0906ed0492cd\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") "
Sep 15 07:58:13.447981 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.447924    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52zzx\" (UniqueName: \"kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx\") pod \"2f065738-983d-49d6-98fb-0906ed0492cd\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") "
Sep 15 07:58:13.447981 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.447970    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs\") pod \"2f065738-983d-49d6-98fb-0906ed0492cd\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") "
Sep 15 07:58:13.448107 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.448014    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls\") pod \"2f065738-983d-49d6-98fb-0906ed0492cd\" (UID: \"2f065738-983d-49d6-98fb-0906ed0492cd\") "
Sep 15 07:58:13.448227 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.448201    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f065738-983d-49d6-98fb-0906ed0492cd" (UID: "2f065738-983d-49d6-98fb-0906ed0492cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:13.448323 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.448304    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f065738-983d-49d6-98fb-0906ed0492cd-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:13.450604 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.450576    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f065738-983d-49d6-98fb-0906ed0492cd" (UID: "2f065738-983d-49d6-98fb-0906ed0492cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:13.450716 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.450654    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx" (OuterVolumeSpecName: "kube-api-access-52zzx") pod "2f065738-983d-49d6-98fb-0906ed0492cd" (UID: "2f065738-983d-49d6-98fb-0906ed0492cd"). InnerVolumeSpecName "kube-api-access-52zzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:13.450810 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.450780    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f065738-983d-49d6-98fb-0906ed0492cd" (UID: "2f065738-983d-49d6-98fb-0906ed0492cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:13.549277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.549235    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52zzx\" (UniqueName: \"kubernetes.io/projected/2f065738-983d-49d6-98fb-0906ed0492cd-kube-api-access-52zzx\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:13.549277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.549270    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:13.549277 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:13.549284    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f065738-983d-49d6-98fb-0906ed0492cd-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:14.401187 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:14.401161    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"
Sep 15 07:58:14.417201 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:14.417167    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:58:14.426791 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:14.426765    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-04464b-f8887f547-2lgxh"]
Sep 15 07:58:16.111204 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:16.111167    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:58:16.378586 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:16.378506    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f065738-983d-49d6-98fb-0906ed0492cd" path="/var/lib/kubelet/pods/2f065738-983d-49d6-98fb-0906ed0492cd/volumes"
Sep 15 07:58:30.712769 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:30.712739    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:58:30.713163 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:30.712975    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" podUID="0201f423-a2d8-4fed-b139-baa22b77424e" containerName="maas-api" containerID="cri-o://bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62" gracePeriod=30
Sep 15 07:58:30.962712 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:30.962684    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:58:31.083287 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083201    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca\") pod \"0201f423-a2d8-4fed-b139-baa22b77424e\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") "
Sep 15 07:58:31.083287 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083261    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs\") pod \"0201f423-a2d8-4fed-b139-baa22b77424e\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") "
Sep 15 07:58:31.083510 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083397    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcdqw\" (UniqueName: \"kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw\") pod \"0201f423-a2d8-4fed-b139-baa22b77424e\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") "
Sep 15 07:58:31.083510 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083441    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") pod \"0201f423-a2d8-4fed-b139-baa22b77424e\" (UID: \"0201f423-a2d8-4fed-b139-baa22b77424e\") "
Sep 15 07:58:31.083626 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083574    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0201f423-a2d8-4fed-b139-baa22b77424e" (UID: "0201f423-a2d8-4fed-b139-baa22b77424e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 07:58:31.083719 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.083695    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0201f423-a2d8-4fed-b139-baa22b77424e-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.085656 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.085629    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0201f423-a2d8-4fed-b139-baa22b77424e" (UID: "0201f423-a2d8-4fed-b139-baa22b77424e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:31.085752 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.085651    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0201f423-a2d8-4fed-b139-baa22b77424e" (UID: "0201f423-a2d8-4fed-b139-baa22b77424e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:31.085872 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.085848    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw" (OuterVolumeSpecName: "kube-api-access-bcdqw") pod "0201f423-a2d8-4fed-b139-baa22b77424e" (UID: "0201f423-a2d8-4fed-b139-baa22b77424e"). InnerVolumeSpecName "kube-api-access-bcdqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:31.184102 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.184069    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.184102 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.184095    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcdqw\" (UniqueName: \"kubernetes.io/projected/0201f423-a2d8-4fed-b139-baa22b77424e-kube-api-access-bcdqw\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.184102 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.184106    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0201f423-a2d8-4fed-b139-baa22b77424e-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:31.493384 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.493350    2574 generic.go:356] "Generic (PLEG): container finished" podID="0201f423-a2d8-4fed-b139-baa22b77424e" containerID="bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62" exitCode=0
Sep 15 07:58:31.493552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.493394    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"
Sep 15 07:58:31.493552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.493427    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" event={"ID":"0201f423-a2d8-4fed-b139-baa22b77424e","Type":"ContainerDied","Data":"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"}
Sep 15 07:58:31.493552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.493481    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf" event={"ID":"0201f423-a2d8-4fed-b139-baa22b77424e","Type":"ContainerDied","Data":"53ded2f18a5386b22b4708dae0a9ba0ce88b520ac30c7725f62beb617776e15a"}
Sep 15 07:58:31.493552 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.493506    2574 scope.go:117] "RemoveContainer" containerID="bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"
Sep 15 07:58:31.503071 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.503053    2574 scope.go:117] "RemoveContainer" containerID="bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"
Sep 15 07:58:31.503343 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:58:31.503326    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62\": container with ID starting with bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62 not found: ID does not exist" containerID="bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"
Sep 15 07:58:31.503416 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.503353    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62"} err="failed to get container status \"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62\": rpc error: code = NotFound desc = could not find container \"bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62\": container with ID starting with bb4c5f7312d139e38301da086d78915029d5607b3c59dbc6a68613cfc44d6d62 not found: ID does not exist"
Sep 15 07:58:31.514597 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.514572    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:58:31.518091 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:31.518070    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e3a482-5c5b5ff597-2ddhf"]
Sep 15 07:58:32.378011 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:32.377977    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0201f423-a2d8-4fed-b139-baa22b77424e" path="/var/lib/kubelet/pods/0201f423-a2d8-4fed-b139-baa22b77424e/volumes"
Sep 15 07:58:33.506800 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:33.506768    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:58:39.737183 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:39.737149    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:58:39.737709 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:39.737385    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" podUID="aa53b6fb-d972-4b11-9179-ce79cf082c80" containerName="manager" containerID="cri-o://37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a" gracePeriod=30
Sep 15 07:58:39.982852 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:39.982828    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:58:40.057483 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.057391    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert\") pod \"aa53b6fb-d972-4b11-9179-ce79cf082c80\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") "
Sep 15 07:58:40.057647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.057516    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xljvq\" (UniqueName: \"kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq\") pod \"aa53b6fb-d972-4b11-9179-ce79cf082c80\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") "
Sep 15 07:58:40.057647 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.057561    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs\") pod \"aa53b6fb-d972-4b11-9179-ce79cf082c80\" (UID: \"aa53b6fb-d972-4b11-9179-ce79cf082c80\") "
Sep 15 07:58:40.059574 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.059542    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq" (OuterVolumeSpecName: "kube-api-access-xljvq") pod "aa53b6fb-d972-4b11-9179-ce79cf082c80" (UID: "aa53b6fb-d972-4b11-9179-ce79cf082c80"). InnerVolumeSpecName "kube-api-access-xljvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 07:58:40.059702 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.059578    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "aa53b6fb-d972-4b11-9179-ce79cf082c80" (UID: "aa53b6fb-d972-4b11-9179-ce79cf082c80"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:40.059940 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.059918    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa53b6fb-d972-4b11-9179-ce79cf082c80" (UID: "aa53b6fb-d972-4b11-9179-ce79cf082c80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 07:58:40.158281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.158251    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xljvq\" (UniqueName: \"kubernetes.io/projected/aa53b6fb-d972-4b11-9179-ce79cf082c80-kube-api-access-xljvq\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:40.158281 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.158277    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:40.158423 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.158290    2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aa53b6fb-d972-4b11-9179-ce79cf082c80-webhook-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 07:58:40.540625 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.540596    2574 generic.go:356] "Generic (PLEG): container finished" podID="aa53b6fb-d972-4b11-9179-ce79cf082c80" containerID="37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a" exitCode=0
Sep 15 07:58:40.540773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.540635    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" event={"ID":"aa53b6fb-d972-4b11-9179-ce79cf082c80","Type":"ContainerDied","Data":"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"}
Sep 15 07:58:40.540773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.540679    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8wlgp" event={"ID":"aa53b6fb-d972-4b11-9179-ce79cf082c80","Type":"ContainerDied","Data":"4904cf6c1d743ebfdcb05e6563b6556820bcd61db66e274e5f857b71f3477c18"}
Sep 15 07:58:40.540773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.540698    2574 scope.go:117] "RemoveContainer" containerID="37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"
Sep 15 07:58:40.540773 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.540638    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8wlgp"
Sep 15 07:58:40.553556 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.553179    2574 scope.go:117] "RemoveContainer" containerID="37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"
Sep 15 07:58:40.553556 ip-10-0-141-227 kubenswrapper[2574]: E0915 07:58:40.553507    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a\": container with ID starting with 37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a not found: ID does not exist" containerID="37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"
Sep 15 07:58:40.553556 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.553541    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a"} err="failed to get container status \"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a\": rpc error: code = NotFound desc = could not find container \"37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a\": container with ID starting with 37edeebc84466226a0e7c98f574799ea7ce3677ef79e8bb3203499f930e4bd4a not found: ID does not exist"
Sep 15 07:58:40.556782 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.556759    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:58:40.560466 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:40.560446    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8wlgp"]
Sep 15 07:58:41.601941 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:41.601863    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 07:58:42.380306 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:58:42.380267    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa53b6fb-d972-4b11-9179-ce79cf082c80" path="/var/lib/kubelet/pods/aa53b6fb-d972-4b11-9179-ce79cf082c80/volumes"
Sep 15 07:59:41.356229 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356191    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356531    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356545    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356558    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356563    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356572    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356577    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356585    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e9f937-8105-45ed-ac90-241b51452bac" containerName="main"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356590    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e9f937-8105-45ed-ac90-241b51452bac" containerName="main"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356597    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa53b6fb-d972-4b11-9179-ce79cf082c80" containerName="manager"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356603    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa53b6fb-d972-4b11-9179-ce79cf082c80" containerName="manager"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356608    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f065738-983d-49d6-98fb-0906ed0492cd" containerName="maas-api"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356613    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f065738-983d-49d6-98fb-0906ed0492cd" containerName="maas-api"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356621    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0201f423-a2d8-4fed-b139-baa22b77424e" containerName="maas-api"
Sep 15 07:59:41.356624 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356626    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0201f423-a2d8-4fed-b139-baa22b77424e" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356634    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356639    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356645    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169902f1-40e3-4c9f-bbab-67709f0583fb" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356659    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="169902f1-40e3-4c9f-bbab-67709f0583fb" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356672    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356678    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356745    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356752    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="156cae51-7ea5-4f2a-b58a-715bd56e93f2" containerName="istio-proxy"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356757    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356763    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="335b69e1-83c0-48dc-932c-5bd1a9df8a37" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356770    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f065738-983d-49d6-98fb-0906ed0492cd" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356776    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0201f423-a2d8-4fed-b139-baa22b77424e" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356782    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa53b6fb-d972-4b11-9179-ce79cf082c80" containerName="manager"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356787    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e9f937-8105-45ed-ac90-241b51452bac" containerName="main"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356793    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc9af7fc-0d54-4b51-a3c9-31f4838cd8f4" containerName="maas-api"
Sep 15 07:59:41.357061 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.356799    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="169902f1-40e3-4c9f-bbab-67709f0583fb" containerName="maas-api"
Sep 15 07:59:41.359562 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.359544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.361788 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.361753    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 07:59:41.362529 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.362500    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 07:59:41.362677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.362534    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vw7xh\""
Sep 15 07:59:41.362677 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.362580    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 07:59:41.370731 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.370707    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 07:59:41.379640 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.379620    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.379738 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.379648    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjgmw\" (UniqueName: \"kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.379738 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.379698    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.480488 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.480435    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.480583 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.480488    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.480583 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.480510    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjgmw\" (UniqueName: \"kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.483109 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.483080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.483207 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.483080    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.487621 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.487600    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjgmw\" (UniqueName: \"kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw\") pod \"maas-controller-656b58fbfb-gkr58\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") " pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.671705 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.671678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:41.944012 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.943986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 07:59:41.945498 ip-10-0-141-227 kubenswrapper[2574]: W0915 07:59:41.945470    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665195b5_1eba_47a6_9b6d_a78cf4f112ab.slice/crio-b6b2d410a2815a3b0dc4369c3a5e805f10a813d1676a177ff2e57a13ec790cdd WatchSource:0}: Error finding container b6b2d410a2815a3b0dc4369c3a5e805f10a813d1676a177ff2e57a13ec790cdd: Status 404 returned error can't find the container with id b6b2d410a2815a3b0dc4369c3a5e805f10a813d1676a177ff2e57a13ec790cdd
Sep 15 07:59:41.946782 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:41.946763    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 07:59:42.868951 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:42.868919    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-gkr58" event={"ID":"665195b5-1eba-47a6-9b6d-a78cf4f112ab","Type":"ContainerStarted","Data":"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"}
Sep 15 07:59:42.869349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:42.868965    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-gkr58" event={"ID":"665195b5-1eba-47a6-9b6d-a78cf4f112ab","Type":"ContainerStarted","Data":"b6b2d410a2815a3b0dc4369c3a5e805f10a813d1676a177ff2e57a13ec790cdd"}
Sep 15 07:59:42.869349 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:42.868986    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 07:59:42.887840 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:42.887792    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b58fbfb-gkr58" podStartSLOduration=1.4722178559999999 podStartE2EDuration="1.887777727s" totalImagesPullingTime="415.559871ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 07:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 07:59:42.886065787 +0000 UTC m=+2470.980232049" watchObservedRunningTime="2026-09-15 07:59:42.887777727 +0000 UTC m=+2470.981943993"
Sep 15 07:59:53.876581 ip-10-0-141-227 kubenswrapper[2574]: I0915 07:59:53.876547    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 08:00:03.111512 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.111479    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:00:03.216423 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.216389    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 08:00:03.216598 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.216564    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-656b58fbfb-gkr58" podUID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" containerName="manager" containerID="cri-o://18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17" gracePeriod=30
Sep 15 08:00:03.457842 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.457817    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 08:00:03.548967 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.548930    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjgmw\" (UniqueName: \"kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw\") pod \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") "
Sep 15 08:00:03.549179 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.549044    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs\") pod \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") "
Sep 15 08:00:03.549179 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.549096    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert\") pod \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\" (UID: \"665195b5-1eba-47a6-9b6d-a78cf4f112ab\") "
Sep 15 08:00:03.551461 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.551397    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "665195b5-1eba-47a6-9b6d-a78cf4f112ab" (UID: "665195b5-1eba-47a6-9b6d-a78cf4f112ab"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:00:03.551619 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.551593    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw" (OuterVolumeSpecName: "kube-api-access-zjgmw") pod "665195b5-1eba-47a6-9b6d-a78cf4f112ab" (UID: "665195b5-1eba-47a6-9b6d-a78cf4f112ab"). InnerVolumeSpecName "kube-api-access-zjgmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 08:00:03.551686 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.551668    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "665195b5-1eba-47a6-9b6d-a78cf4f112ab" (UID: "665195b5-1eba-47a6-9b6d-a78cf4f112ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:00:03.650057 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.649976    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:00:03.650057 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.650002    2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/665195b5-1eba-47a6-9b6d-a78cf4f112ab-webhook-cert\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:00:03.650057 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.650011    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjgmw\" (UniqueName: \"kubernetes.io/projected/665195b5-1eba-47a6-9b6d-a78cf4f112ab-kube-api-access-zjgmw\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:00:03.984435 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.984389    2574 generic.go:356] "Generic (PLEG): container finished" podID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" containerID="18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17" exitCode=0
Sep 15 08:00:03.984708 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.984449    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-gkr58"
Sep 15 08:00:03.984708 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.984442    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-gkr58" event={"ID":"665195b5-1eba-47a6-9b6d-a78cf4f112ab","Type":"ContainerDied","Data":"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"}
Sep 15 08:00:03.984708 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.984594    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-gkr58" event={"ID":"665195b5-1eba-47a6-9b6d-a78cf4f112ab","Type":"ContainerDied","Data":"b6b2d410a2815a3b0dc4369c3a5e805f10a813d1676a177ff2e57a13ec790cdd"}
Sep 15 08:00:03.984708 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.984632    2574 scope.go:117] "RemoveContainer" containerID="18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"
Sep 15 08:00:03.996215 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.996107    2574 scope.go:117] "RemoveContainer" containerID="18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"
Sep 15 08:00:03.996552 ip-10-0-141-227 kubenswrapper[2574]: E0915 08:00:03.996526    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17\": container with ID starting with 18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17 not found: ID does not exist" containerID="18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"
Sep 15 08:00:03.996647 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:03.996562    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17"} err="failed to get container status \"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17\": rpc error: code = NotFound desc = could not find container \"18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17\": container with ID starting with 18985997e5f9efa26e89561366b2d806636565ee69a5cfefc55b5dcf0c0a9d17 not found: ID does not exist"
Sep 15 08:00:04.009467 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.009437    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 08:00:04.012678 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.012654    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-gkr58"]
Sep 15 08:00:04.378761 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.378682    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" path="/var/lib/kubelet/pods/665195b5-1eba-47a6-9b6d-a78cf4f112ab/volumes"
Sep 15 08:00:04.780176 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780143    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8bwsq"]
Sep 15 08:00:04.780485 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780473    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" containerName="manager"
Sep 15 08:00:04.780535 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780487    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" containerName="manager"
Sep 15 08:00:04.780535 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780499    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 08:00:04.780535 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780504    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 08:00:04.780640 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780568    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d40bbc7-19c9-478f-bf92-99ab2afd1a30" containerName="istio-proxy"
Sep 15 08:00:04.780640 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.780578    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="665195b5-1eba-47a6-9b6d-a78cf4f112ab" containerName="manager"
Sep 15 08:00:04.784773 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.784755    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.787427 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.787404    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 08:00:04.788009 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.787978    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 08:00:04.788009 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.788005    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 08:00:04.788173 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.787985    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-vw7xh\""
Sep 15 08:00:04.796832 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.796807    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8bwsq"]
Sep 15 08:00:04.862236 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.862197    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-metrics-certs\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.862416 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.862302    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-webhook-cert\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.862416 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.862356    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tchl\" (UniqueName: \"kubernetes.io/projected/94ec87e2-035a-473b-a6e9-5da8c4c349f9-kube-api-access-7tchl\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.963444 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.963410    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-metrics-certs\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.963665 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.963468    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-webhook-cert\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.963665 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.963490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tchl\" (UniqueName: \"kubernetes.io/projected/94ec87e2-035a-473b-a6e9-5da8c4c349f9-kube-api-access-7tchl\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.966079 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.966047    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-webhook-cert\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.966079 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.966069    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94ec87e2-035a-473b-a6e9-5da8c4c349f9-metrics-certs\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:04.972457 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:04.972433    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tchl\" (UniqueName: \"kubernetes.io/projected/94ec87e2-035a-473b-a6e9-5da8c4c349f9-kube-api-access-7tchl\") pod \"maas-controller-656b58fbfb-8bwsq\" (UID: \"94ec87e2-035a-473b-a6e9-5da8c4c349f9\") " pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:05.095054 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:05.094964    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:05.375200 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:05.375153    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-656b58fbfb-8bwsq"]
Sep 15 08:00:05.375647 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:00:05.375623    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ec87e2_035a_473b_a6e9_5da8c4c349f9.slice/crio-872335337ca2de469dd30514faa4ab1e9b97396821cf104d6cf1b97acfcbc618 WatchSource:0}: Error finding container 872335337ca2de469dd30514faa4ab1e9b97396821cf104d6cf1b97acfcbc618: Status 404 returned error can't find the container with id 872335337ca2de469dd30514faa4ab1e9b97396821cf104d6cf1b97acfcbc618
Sep 15 08:00:05.998791 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:05.998692    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8bwsq" event={"ID":"94ec87e2-035a-473b-a6e9-5da8c4c349f9","Type":"ContainerStarted","Data":"1df38c758cd0d896580d121f2e4cc945e6f8aea745ebf06da6bc0db8de312bef"}
Sep 15 08:00:05.998791 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:05.998749    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-656b58fbfb-8bwsq" event={"ID":"94ec87e2-035a-473b-a6e9-5da8c4c349f9","Type":"ContainerStarted","Data":"872335337ca2de469dd30514faa4ab1e9b97396821cf104d6cf1b97acfcbc618"}
Sep 15 08:00:05.998791 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:05.998777    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:06.014882 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:06.014816    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-656b58fbfb-8bwsq" podStartSLOduration=1.6661757499999998 podStartE2EDuration="2.014795833s" totalImagesPullingTime="348.620083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:00:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:00:06.013581009 +0000 UTC m=+2494.107747283" watchObservedRunningTime="2026-09-15 08:00:06.014795833 +0000 UTC m=+2494.108962096"
Sep 15 08:00:17.006360 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:17.006329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-656b58fbfb-8bwsq"
Sep 15 08:00:27.005779 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:27.005694    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:00:34.706655 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:34.706613    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:00:43.008520 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:00:43.008480    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:01:01.309081 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:01.309042    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:01:07.300855 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:07.300821    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:01:08.997263 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:08.997226    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 08:01:08.997749 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:08.997487    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" podUID="8d098681-6dc5-46de-946f-71470037c841" containerName="manager" containerID="cri-o://c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f" gracePeriod=10
Sep 15 08:01:11.133799 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.133772    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 08:01:11.272262 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.272179    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume\") pod \"8d098681-6dc5-46de-946f-71470037c841\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") "
Sep 15 08:01:11.272406 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.272269    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4v8l\" (UniqueName: \"kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l\") pod \"8d098681-6dc5-46de-946f-71470037c841\" (UID: \"8d098681-6dc5-46de-946f-71470037c841\") "
Sep 15 08:01:11.272490 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.272468    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8d098681-6dc5-46de-946f-71470037c841" (UID: "8d098681-6dc5-46de-946f-71470037c841"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:01:11.274292 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.274264    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l" (OuterVolumeSpecName: "kube-api-access-z4v8l") pod "8d098681-6dc5-46de-946f-71470037c841" (UID: "8d098681-6dc5-46de-946f-71470037c841"). InnerVolumeSpecName "kube-api-access-z4v8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 08:01:11.347815 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.347783    2574 generic.go:356] "Generic (PLEG): container finished" podID="8d098681-6dc5-46de-946f-71470037c841" containerID="c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f" exitCode=0
Sep 15 08:01:11.347954 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.347822    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"
Sep 15 08:01:11.347954 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.347871    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" event={"ID":"8d098681-6dc5-46de-946f-71470037c841","Type":"ContainerDied","Data":"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"}
Sep 15 08:01:11.347954 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.347924    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp" event={"ID":"8d098681-6dc5-46de-946f-71470037c841","Type":"ContainerDied","Data":"4d2a06ff66cad3f30d246ecede2d5713bbe43333517cbdeceb73a9959a82e766"}
Sep 15 08:01:11.347954 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.347951    2574 scope.go:117] "RemoveContainer" containerID="c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"
Sep 15 08:01:11.358318 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.358303    2574 scope.go:117] "RemoveContainer" containerID="c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"
Sep 15 08:01:11.358594 ip-10-0-141-227 kubenswrapper[2574]: E0915 08:01:11.358574    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f\": container with ID starting with c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f not found: ID does not exist" containerID="c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"
Sep 15 08:01:11.358647 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.358602    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f"} err="failed to get container status \"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f\": rpc error: code = NotFound desc = could not find container \"c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f\": container with ID starting with c2cce2f98c5ec05716a10aa93b3681a646776cd9e649f65748dc2bcf0087093f not found: ID does not exist"
Sep 15 08:01:11.373759 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.373732    2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8d098681-6dc5-46de-946f-71470037c841-extensions-socket-volume\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:01:11.373911 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.373857    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4v8l\" (UniqueName: \"kubernetes.io/projected/8d098681-6dc5-46de-946f-71470037c841-kube-api-access-z4v8l\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:01:11.374016 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.373900    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 08:01:11.377231 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:11.377206    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-4jvrp"]
Sep 15 08:01:12.377781 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:12.377746    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d098681-6dc5-46de-946f-71470037c841" path="/var/lib/kubelet/pods/8d098681-6dc5-46de-946f-71470037c841/volumes"
Sep 15 08:01:32.887438 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:01:32.887402    2574 scope.go:117] "RemoveContainer" containerID="ff9918ec86b500ea6c113e2ac06e8db2abec39f9b4b995f6801a8e1cfed74bee"
Sep 15 08:02:09.726998 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.726921    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"]
Sep 15 08:02:09.727463 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.727442    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d098681-6dc5-46de-946f-71470037c841" containerName="manager"
Sep 15 08:02:09.727509 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.727466    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d098681-6dc5-46de-946f-71470037c841" containerName="manager"
Sep 15 08:02:09.727570 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.727559    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d098681-6dc5-46de-946f-71470037c841" containerName="manager"
Sep 15 08:02:09.730881 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.730865    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.733271 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.733246    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5dx5m\""
Sep 15 08:02:09.741305 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.741283    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"]
Sep 15 08:02:09.890849 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.890816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.890849 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.890848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cll6\" (UniqueName: \"kubernetes.io/projected/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-kube-api-access-4cll6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.991463 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.991400    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.991463 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.991428    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cll6\" (UniqueName: \"kubernetes.io/projected/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-kube-api-access-4cll6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.991742 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.991723    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:09.998622 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:09.998603    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cll6\" (UniqueName: \"kubernetes.io/projected/e307cef6-bbf3-496a-a8b7-58ebba9cfe87-kube-api-access-4cll6\") pod \"kuadrant-operator-controller-manager-5769f99d9f-qgxpl\" (UID: \"e307cef6-bbf3-496a-a8b7-58ebba9cfe87\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:10.042424 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.042395    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:10.321228 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.321200    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"]
Sep 15 08:02:10.322790 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:02:10.322762    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode307cef6_bbf3_496a_a8b7_58ebba9cfe87.slice/crio-c3ab2d9b7b386c7a0c3f6eba3a01a6b50b1fd9db014a4f630848b4510f152934 WatchSource:0}: Error finding container c3ab2d9b7b386c7a0c3f6eba3a01a6b50b1fd9db014a4f630848b4510f152934: Status 404 returned error can't find the container with id c3ab2d9b7b386c7a0c3f6eba3a01a6b50b1fd9db014a4f630848b4510f152934
Sep 15 08:02:10.655229 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.655144    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl" event={"ID":"e307cef6-bbf3-496a-a8b7-58ebba9cfe87","Type":"ContainerStarted","Data":"89cafcb25c40a635ec284ce0bf0c093bb29d53873a3223c1706e2731b812e194"}
Sep 15 08:02:10.655229 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.655214    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl" event={"ID":"e307cef6-bbf3-496a-a8b7-58ebba9cfe87","Type":"ContainerStarted","Data":"c3ab2d9b7b386c7a0c3f6eba3a01a6b50b1fd9db014a4f630848b4510f152934"}
Sep 15 08:02:10.655403 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.655243    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:10.675290 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:10.675238    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl" podStartSLOduration=1.6752196499999998 podStartE2EDuration="1.67521965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:02:10.67358316 +0000 UTC m=+2618.767749424" watchObservedRunningTime="2026-09-15 08:02:10.67521965 +0000 UTC m=+2618.769385913"
Sep 15 08:02:21.667958 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:21.667918    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-qgxpl"
Sep 15 08:02:33.602442 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:33.602401    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:02:42.115240 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:42.115196    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:02:47.159483 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.159448    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:02:47.162712 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.162691    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.167667 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.167621    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 08:02:47.168398 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.168377    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 08:02:47.168555 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.168529    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-3a699cc7-kserve-self-signed-certs\""
Sep 15 08:02:47.168555 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.168511    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-84rbz\""
Sep 15 08:02:47.183879 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.183858    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:02:47.289089 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289060    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr59s\" (UniqueName: \"kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.289279 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289098    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.289279 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289198    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.289279 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289267    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.289459 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.289459 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.289385    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390204 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390168    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390339 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390215    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390403 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390335    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390465 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390409    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390523 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390470    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr59s\" (UniqueName: \"kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390523 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390638 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390758 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390727    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.390823 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.390758    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.392423 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.392393    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.392860 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.392840    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.405019 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.404996    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr59s\" (UniqueName: \"kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s\") pod \"e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") " pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.471913 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.471854    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:47.746504 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.746478    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:02:47.747783 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:02:47.747750    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c4dfbab_1a7d_4900_880b_5a29a7fbc00a.slice/crio-9a79c284fd5100eda5a4fadaaf62576afd68fc82c8124a8ba8687792f50a3968 WatchSource:0}: Error finding container 9a79c284fd5100eda5a4fadaaf62576afd68fc82c8124a8ba8687792f50a3968: Status 404 returned error can't find the container with id 9a79c284fd5100eda5a4fadaaf62576afd68fc82c8124a8ba8687792f50a3968
Sep 15 08:02:47.870097 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.870058    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" event={"ID":"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a","Type":"ContainerStarted","Data":"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"}
Sep 15 08:02:47.870263 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.870109    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" event={"ID":"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a","Type":"ContainerStarted","Data":"9a79c284fd5100eda5a4fadaaf62576afd68fc82c8124a8ba8687792f50a3968"}
Sep 15 08:02:47.891103 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:47.891054    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" podStartSLOduration=0.891038931 podStartE2EDuration="891.038931ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:02:47.889499269 +0000 UTC m=+2655.983665553" watchObservedRunningTime="2026-09-15 08:02:47.891038931 +0000 UTC m=+2655.985205193"
Sep 15 08:02:49.135718 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:49.135679    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:02:51.901415 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:51.901373    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:02:51.904641 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:51.904613    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:51.907034 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:51.907010    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3a699cc7-kserve-self-signed-certs\""
Sep 15 08:02:51.916045 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:51.916024    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:02:52.032753 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032723    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.032897 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032766    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.032897 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032830    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npsx9\" (UniqueName: \"kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.033010 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.033010 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032930    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.033010 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.032996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134044 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134010    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134196 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134074    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npsx9\" (UniqueName: \"kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134196 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134104    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134196 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134152    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134328 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134328 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134250    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134550 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134523    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134633 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134530    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.134633 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.134606    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.136575 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.136553    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.136809 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.136792    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.141875 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.141857    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npsx9\" (UniqueName: \"kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9\") pod \"e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") " pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.215040 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.215018    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:02:52.490861 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.490833    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:02:52.493381 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:02:52.493356    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6a13648_eb08_4ea8_8282_6d4a11d9b8b0.slice/crio-125022533ed3d5774c6764e596ea7292d218a3ed9766822d3f2dbee2238a0973 WatchSource:0}: Error finding container 125022533ed3d5774c6764e596ea7292d218a3ed9766822d3f2dbee2238a0973: Status 404 returned error can't find the container with id 125022533ed3d5774c6764e596ea7292d218a3ed9766822d3f2dbee2238a0973
Sep 15 08:02:52.898398 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.898310    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" event={"ID":"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0","Type":"ContainerStarted","Data":"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"}
Sep 15 08:02:52.898398 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.898361    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" event={"ID":"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0","Type":"ContainerStarted","Data":"125022533ed3d5774c6764e596ea7292d218a3ed9766822d3f2dbee2238a0973"}
Sep 15 08:02:52.915467 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:52.915417    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" podStartSLOduration=1.9154025080000001 podStartE2EDuration="1.915402508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:02:52.914251328 +0000 UTC m=+2661.008417611" watchObservedRunningTime="2026-09-15 08:02:52.915402508 +0000 UTC m=+2661.009568772"
Sep 15 08:02:57.472683 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:57.472648    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:57.473087 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:57.472700    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:57.484922 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:57.484898    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:02:57.935581 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:02:57.935559    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:03:01.533925 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:01.533890    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:03:02.216046 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:02.215999    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:03:02.216247 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:02.216057    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:03:02.228886 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:02.228860    2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:03:02.966354 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:02.966329    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:03:44.536894 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:44.536852    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:03:46.624220 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:46.624184    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:03:46.711253 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:46.711223    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:03:55.807724 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:55.807687    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:03:59.561369 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.561315    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" podUID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" containerName="main" containerID="cri-o://7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67" gracePeriod=45
Sep 15 08:03:59.803530 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.803505    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:03:59.960266 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960238    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr59s\" (UniqueName: \"kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960395 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960277    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960395 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960295    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960395 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960364    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960395 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960385    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960580 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960399    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home\") pod \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\" (UID: \"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a\") "
Sep 15 08:03:59.960580 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960529    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:03:59.960707 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960595    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache" (OuterVolumeSpecName: "model-cache") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:03:59.960707 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960655    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:03:59.960810 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.960761    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home" (OuterVolumeSpecName: "home") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:03:59.962525 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.962496    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:03:59.962525 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.962519    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm" (OuterVolumeSpecName: "dshm") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:03:59.962678 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:03:59.962543    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s" (OuterVolumeSpecName: "kube-api-access-dr59s") pod "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" (UID: "0c4dfbab-1a7d-4900-880b-5a29a7fbc00a"). InnerVolumeSpecName "kube-api-access-dr59s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 08:04:00.061982 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.061943    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:00.061982 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.061974    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:00.062224 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.061990    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr59s\" (UniqueName: \"kubernetes.io/projected/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-kube-api-access-dr59s\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:00.062224 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.062005    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:00.062224 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.062017    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:00.252899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.252809    2574 generic.go:356] "Generic (PLEG): container finished" podID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" containerID="7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67" exitCode=0
Sep 15 08:04:00.252899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.252849    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"
Sep 15 08:04:00.253131 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.252897    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" event={"ID":"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a","Type":"ContainerDied","Data":"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"}
Sep 15 08:04:00.253131 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.252939    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm" event={"ID":"0c4dfbab-1a7d-4900-880b-5a29a7fbc00a","Type":"ContainerDied","Data":"9a79c284fd5100eda5a4fadaaf62576afd68fc82c8124a8ba8687792f50a3968"}
Sep 15 08:04:00.253131 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.252962    2574 scope.go:117] "RemoveContainer" containerID="7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"
Sep 15 08:04:00.263018 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.263006    2574 scope.go:117] "RemoveContainer" containerID="7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"
Sep 15 08:04:00.263314 ip-10-0-141-227 kubenswrapper[2574]: E0915 08:04:00.263295    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67\": container with ID starting with 7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67 not found: ID does not exist" containerID="7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"
Sep 15 08:04:00.263404 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.263324    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67"} err="failed to get container status \"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67\": rpc error: code = NotFound desc = could not find container \"7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67\": container with ID starting with 7a840ccc6b5b255d20e03596414082885bc78dbcb704f972c20578be11007d67 not found: ID does not exist"
Sep 15 08:04:00.275809 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.275787    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:04:00.280656 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.280622    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-3a699cc7-kserve-6c9488d57f-nbgmm"]
Sep 15 08:04:00.378898 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:00.378865    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" path="/var/lib/kubelet/pods/0c4dfbab-1a7d-4900-880b-5a29a7fbc00a/volumes"
Sep 15 08:04:01.643060 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.642982    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" podUID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" containerName="main" containerID="cri-o://b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40" gracePeriod=45
Sep 15 08:04:01.883030 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.883007    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:04:01.976503 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976470    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976672 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976672 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976588    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976672 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976603    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976672 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976631    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976694    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npsx9\" (UniqueName: \"kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9\") pod \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\" (UID: \"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0\") "
Sep 15 08:04:01.976899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976856    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home" (OuterVolumeSpecName: "home") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:04:01.976899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976869    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:04:01.977054 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.976926    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:04:01.977054 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.977042    2574 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-home\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:01.977171 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.977061    2574 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tmp-dir\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:01.977171 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.977077    2574 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-model-cache\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:01.978573 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.978547    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm" (OuterVolumeSpecName: "dshm") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 08:04:01.978736 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.978715    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:04:01.978818 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:01.978802    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9" (OuterVolumeSpecName: "kube-api-access-npsx9") pod "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" (UID: "c6a13648-eb08-4ea8-8282-6d4a11d9b8b0"). InnerVolumeSpecName "kube-api-access-npsx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 08:04:02.077944 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.077921    2574 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-tls-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:02.077944 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.077943    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npsx9\" (UniqueName: \"kubernetes.io/projected/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-kube-api-access-npsx9\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:02.078081 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.077952    2574 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0-dshm\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:02.268676 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.268593    2574 generic.go:356] "Generic (PLEG): container finished" podID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" containerID="b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40" exitCode=0
Sep 15 08:04:02.268676 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.268636    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"
Sep 15 08:04:02.268837 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.268682    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" event={"ID":"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0","Type":"ContainerDied","Data":"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"}
Sep 15 08:04:02.268837 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.268723    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp" event={"ID":"c6a13648-eb08-4ea8-8282-6d4a11d9b8b0","Type":"ContainerDied","Data":"125022533ed3d5774c6764e596ea7292d218a3ed9766822d3f2dbee2238a0973"}
Sep 15 08:04:02.268837 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.268744    2574 scope.go:117] "RemoveContainer" containerID="b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"
Sep 15 08:04:02.278600 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.278581    2574 scope.go:117] "RemoveContainer" containerID="b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"
Sep 15 08:04:02.278866 ip-10-0-141-227 kubenswrapper[2574]: E0915 08:04:02.278842    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40\": container with ID starting with b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40 not found: ID does not exist" containerID="b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"
Sep 15 08:04:02.278952 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.278872    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40"} err="failed to get container status \"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40\": rpc error: code = NotFound desc = could not find container \"b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40\": container with ID starting with b1513f83a1cd29b511ab416798f316545edb8f71cfe5d0be7b40de74835a5a40 not found: ID does not exist"
Sep 15 08:04:02.291468 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.291446    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:04:02.297945 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.297926    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3a699cc7-kserve-7b5fdcb58c-wwpdp"]
Sep 15 08:04:02.379628 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:02.379605    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" path="/var/lib/kubelet/pods/c6a13648-eb08-4ea8-8282-6d4a11d9b8b0/volumes"
Sep 15 08:04:03.007003 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:03.006964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:09.504363 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.504324    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:09.651580 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.651547    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"]
Sep 15 08:04:09.651972 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.651954    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" containerName="main"
Sep 15 08:04:09.652085 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.651974    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" containerName="main"
Sep 15 08:04:09.652085 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.652001    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" containerName="main"
Sep 15 08:04:09.652085 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.652012    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" containerName="main"
Sep 15 08:04:09.652274 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.652096    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6a13648-eb08-4ea8-8282-6d4a11d9b8b0" containerName="main"
Sep 15 08:04:09.652274 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.652110    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4dfbab-1a7d-4900-880b-5a29a7fbc00a" containerName="main"
Sep 15 08:04:09.656373 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.656352    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.665365 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.665338    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"]
Sep 15 08:04:09.834158 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.834064    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-metrics-certs\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.834158 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.834107    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-openshift-service-ca\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.834341 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.834162    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks2d4\" (UniqueName: \"kubernetes.io/projected/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-kube-api-access-ks2d4\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.834341 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.834186    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-maas-api-tls\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.934727 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.934690    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-metrics-certs\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.934903 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.934748    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-openshift-service-ca\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.934903 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.934776    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks2d4\" (UniqueName: \"kubernetes.io/projected/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-kube-api-access-ks2d4\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.934903 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.934805    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-maas-api-tls\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.935516 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.935487    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-openshift-service-ca\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.937570 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.937543    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-metrics-certs\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.937982 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.937966    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-maas-api-tls\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.942478 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.942455    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks2d4\" (UniqueName: \"kubernetes.io/projected/8c8e40bc-e6b5-48ad-a3b7-bde16a13b012-kube-api-access-ks2d4\") pod \"maas-api-6b6696b464-npqrs\" (UID: \"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012\") " pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:09.970049 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:09.970025    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:10.243577 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:10.243552    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"]
Sep 15 08:04:10.245281 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:04:10.245252    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8e40bc_e6b5_48ad_a3b7_bde16a13b012.slice/crio-7b80c458cfde11a6a09169db376306239bc3eb62f29b1a048408e1da5c98bf88 WatchSource:0}: Error finding container 7b80c458cfde11a6a09169db376306239bc3eb62f29b1a048408e1da5c98bf88: Status 404 returned error can't find the container with id 7b80c458cfde11a6a09169db376306239bc3eb62f29b1a048408e1da5c98bf88
Sep 15 08:04:10.311681 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:10.311646    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs" event={"ID":"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012","Type":"ContainerStarted","Data":"7b80c458cfde11a6a09169db376306239bc3eb62f29b1a048408e1da5c98bf88"}
Sep 15 08:04:11.318630 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:11.318589    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs" event={"ID":"8c8e40bc-e6b5-48ad-a3b7-bde16a13b012","Type":"ContainerStarted","Data":"0339d6af8a803113573603c52ab15bdcf5c0f15b8863af9f1548139d4107e27d"}
Sep 15 08:04:11.319013 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:11.318699    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:11.335282 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:11.335234    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs" podStartSLOduration=1.912548892 podStartE2EDuration="2.335218675s" totalImagesPullingTime="422.669783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:04:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:04:11.333614761 +0000 UTC m=+2739.427781023" watchObservedRunningTime="2026-09-15 08:04:11.335218675 +0000 UTC m=+2739.429384979"
Sep 15 08:04:17.223820 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:17.223784    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:17.328753 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:17.328726    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b6696b464-npqrs"
Sep 15 08:04:24.119422 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:24.119384    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:24.925610 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:24.925577    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 08:04:24.925951 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:24.925889    2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" podUID="9f504830-d566-40d4-b037-4823b074978e" containerName="maas-api" containerID="cri-o://fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b" gracePeriod=30
Sep 15 08:04:25.546471 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:25.546428    2574 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" podUID="9f504830-d566-40d4-b037-4823b074978e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused"
Sep 15 08:04:26.168020 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.167996    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 08:04:26.258473 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.258404    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxn59\" (UniqueName: \"kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59\") pod \"9f504830-d566-40d4-b037-4823b074978e\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") "
Sep 15 08:04:26.258622 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.258474    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs\") pod \"9f504830-d566-40d4-b037-4823b074978e\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") "
Sep 15 08:04:26.258622 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.258492    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls\") pod \"9f504830-d566-40d4-b037-4823b074978e\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") "
Sep 15 08:04:26.258622 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.258523    2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca\") pod \"9f504830-d566-40d4-b037-4823b074978e\" (UID: \"9f504830-d566-40d4-b037-4823b074978e\") "
Sep 15 08:04:26.258964 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.258934    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9f504830-d566-40d4-b037-4823b074978e" (UID: "9f504830-d566-40d4-b037-4823b074978e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 08:04:26.260524 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.260493    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59" (OuterVolumeSpecName: "kube-api-access-lxn59") pod "9f504830-d566-40d4-b037-4823b074978e" (UID: "9f504830-d566-40d4-b037-4823b074978e"). InnerVolumeSpecName "kube-api-access-lxn59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 08:04:26.260624 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.260531    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9f504830-d566-40d4-b037-4823b074978e" (UID: "9f504830-d566-40d4-b037-4823b074978e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:04:26.260624 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.260611    2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9f504830-d566-40d4-b037-4823b074978e" (UID: "9f504830-d566-40d4-b037-4823b074978e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 08:04:26.359689 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.359649    2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxn59\" (UniqueName: \"kubernetes.io/projected/9f504830-d566-40d4-b037-4823b074978e-kube-api-access-lxn59\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:26.359689 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.359686    2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-metrics-certs\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:26.359828 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.359696    2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9f504830-d566-40d4-b037-4823b074978e-maas-api-tls\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:26.359828 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.359705    2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f504830-d566-40d4-b037-4823b074978e-openshift-service-ca\") on node \"ip-10-0-141-227.ec2.internal\" DevicePath \"\""
Sep 15 08:04:26.401493 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.401466    2574 generic.go:356] "Generic (PLEG): container finished" podID="9f504830-d566-40d4-b037-4823b074978e" containerID="fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b" exitCode=0
Sep 15 08:04:26.401627 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.401516    2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"
Sep 15 08:04:26.401627 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.401548    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" event={"ID":"9f504830-d566-40d4-b037-4823b074978e","Type":"ContainerDied","Data":"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"}
Sep 15 08:04:26.401627 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.401610    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc" event={"ID":"9f504830-d566-40d4-b037-4823b074978e","Type":"ContainerDied","Data":"ac040b95d1f8eca6108e8f50417dd8ece1b6fb3b7a98ef0db7f5d29735d2f4a9"}
Sep 15 08:04:26.401761 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.401631    2574 scope.go:117] "RemoveContainer" containerID="fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"
Sep 15 08:04:26.410426 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.410409    2574 scope.go:117] "RemoveContainer" containerID="fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"
Sep 15 08:04:26.410677 ip-10-0-141-227 kubenswrapper[2574]: E0915 08:04:26.410661    2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b\": container with ID starting with fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b not found: ID does not exist" containerID="fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"
Sep 15 08:04:26.410726 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.410683    2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b"} err="failed to get container status \"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b\": rpc error: code = NotFound desc = could not find container \"fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b\": container with ID starting with fbce5daa741332c2c3881e8b2a6b76ae2ba08b14f1bc1fb73f1ab54a1095870b not found: ID does not exist"
Sep 15 08:04:26.417540 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.417521    2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 08:04:26.424163 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:26.424144    2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-66bb9979b8-2xntc"]
Sep 15 08:04:28.377899 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:28.377864    2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f504830-d566-40d4-b037-4823b074978e" path="/var/lib/kubelet/pods/9f504830-d566-40d4-b037-4823b074978e/volumes"
Sep 15 08:04:29.809849 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:29.809814    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:37.212057 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:37.212021    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-26rq6"]
Sep 15 08:04:39.454104 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:39.454065    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75bcc56485-7tvtl_e886b953-6c7c-4127-8569-cf75a55ecba8/authorino/0.log"
Sep 15 08:04:44.876007 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:44.875972    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-smrhn_cd7c9f17-47bc-4550-a393-c5aaa46943ee/manager/0.log"
Sep 15 08:04:45.015482 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:45.015452    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q29mg_54dbf0d5-437a-4355-8e8d-c2915d3aaa59/manager/0.log"
Sep 15 08:04:45.147070 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:45.147005    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-656b58fbfb-8bwsq_94ec87e2-035a-473b-a6e9-5da8c4c349f9/manager/0.log"
Sep 15 08:04:45.412881 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:45.412858    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bkj9r_5e58c92a-eecb-4909-bf1b-b00b4059e4b9/manager/0.log"
Sep 15 08:04:45.820099 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:45.820031    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-tsjj8_a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520/copy-manifests/0.log"
Sep 15 08:04:45.825967 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:45.825943    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-tsjj8_a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520/manager/0.log"
Sep 15 08:04:46.153863 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:46.153797    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b6696b464-npqrs_8c8e40bc-e6b5-48ad-a3b7-bde16a13b012/maas-api/0.log"
Sep 15 08:04:46.614696 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:46.614660    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-rjvd6_65eff245-4884-478f-bc0c-994bc46f6791/postgres/0.log"
Sep 15 08:04:47.422531 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:47.422498    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75bcc56485-7tvtl_e886b953-6c7c-4127-8569-cf75a55ecba8/authorino/0.log"
Sep 15 08:04:47.553820 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:47.553796    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cx79r_c48d52a2-e97c-4db6-b960-c48cf826201c/manager/0.log"
Sep 15 08:04:47.686741 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:47.686683    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-r9tjx_08aeeb1f-4086-4b9e-84df-61f6d93a3534/manager/0.log"
Sep 15 08:04:47.950912 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:47.950855    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-qgxpl_e307cef6-bbf3-496a-a8b7-58ebba9cfe87/manager/0.log"
Sep 15 08:04:48.067799 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:48.067777    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-26rq6_7ac85e8e-0767-4b03-b51f-5f536c71d6dd/limitador/0.log"
Sep 15 08:04:48.892014 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:48.891984    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-v8gdn_3536fa4f-d8f7-41e4-b371-a0687e74f611/kube-auth-proxy/0.log"
Sep 15 08:04:49.008851 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:49.008822    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/1.log"
Sep 15 08:04:55.596850 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.596771    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c8zpm/must-gather-wdg9d"]
Sep 15 08:04:55.597559 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.597127    2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f504830-d566-40d4-b037-4823b074978e" containerName="maas-api"
Sep 15 08:04:55.597559 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.597140    2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f504830-d566-40d4-b037-4823b074978e" containerName="maas-api"
Sep 15 08:04:55.597559 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.597218    2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f504830-d566-40d4-b037-4823b074978e" containerName="maas-api"
Sep 15 08:04:55.600387 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.600370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.602840 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.602819    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c8zpm\"/\"kube-root-ca.crt\""
Sep 15 08:04:55.602962 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.602824    2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c8zpm\"/\"openshift-service-ca.crt\""
Sep 15 08:04:55.603341 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.603327    2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c8zpm\"/\"default-dockercfg-5fl5x\""
Sep 15 08:04:55.609080 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.609058    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/must-gather-wdg9d"]
Sep 15 08:04:55.687270 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.687240    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6r7\" (UniqueName: \"kubernetes.io/projected/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-kube-api-access-5l6r7\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.687380 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.687357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-must-gather-output\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.788419 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.788391    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-must-gather-output\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.788516 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.788431    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6r7\" (UniqueName: \"kubernetes.io/projected/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-kube-api-access-5l6r7\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.788723 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.788706    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-must-gather-output\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.795809 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.795779    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6r7\" (UniqueName: \"kubernetes.io/projected/d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad-kube-api-access-5l6r7\") pod \"must-gather-wdg9d\" (UID: \"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad\") " pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:55.910742 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:55.910716    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/must-gather-wdg9d"
Sep 15 08:04:56.184992 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:56.184964    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/must-gather-wdg9d"]
Sep 15 08:04:56.186849 ip-10-0-141-227 kubenswrapper[2574]: W0915 08:04:56.186826    2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6f3d01d_cdc7_4a50_b191_7ae5d0e408ad.slice/crio-fa71cc8d21f633c27cf175ae538a54f9f05ff66a5e24ac281d601eacf691785f WatchSource:0}: Error finding container fa71cc8d21f633c27cf175ae538a54f9f05ff66a5e24ac281d601eacf691785f: Status 404 returned error can't find the container with id fa71cc8d21f633c27cf175ae538a54f9f05ff66a5e24ac281d601eacf691785f
Sep 15 08:04:56.189065 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:56.189046    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 08:04:56.563622 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:56.563556    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/must-gather-wdg9d" event={"ID":"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad","Type":"ContainerStarted","Data":"fa71cc8d21f633c27cf175ae538a54f9f05ff66a5e24ac281d601eacf691785f"}
Sep 15 08:04:57.572978 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:57.572937    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/must-gather-wdg9d" event={"ID":"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad","Type":"ContainerStarted","Data":"dff7eb4504595b90657cdc32134027c343a11af6e2f13257913fc79c60b1534c"}
Sep 15 08:04:57.573483 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:57.572992    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/must-gather-wdg9d" event={"ID":"d6f3d01d-cdc7-4a50-b191-7ae5d0e408ad","Type":"ContainerStarted","Data":"d6d2a99836713e37606146da0f7bec332100fdca38fe173cc93d7abae7053073"}
Sep 15 08:04:57.591523 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:57.591462    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c8zpm/must-gather-wdg9d" podStartSLOduration=1.801819451 podStartE2EDuration="2.591442775s" totalImagesPullingTime="789.623324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 08:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:04:57.589022959 +0000 UTC m=+2785.683189222" watchObservedRunningTime="2026-09-15 08:04:57.591442775 +0000 UTC m=+2785.685609051"
Sep 15 08:04:58.515719 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:58.515684    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6rthp_1d5ee397-08ec-4837-a81f-3e22caddadcc/global-pull-secret-syncer/0.log"
Sep 15 08:04:58.680371 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:58.680338    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-b8f8q_6af4b9b5-9713-46ae-bfd0-22c7ddf241bb/konnectivity-agent/0.log"
Sep 15 08:04:58.749564 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:04:58.749532    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-227.ec2.internal_d9f17e721a455587375069a20c2a5ed0/haproxy/0.log"
Sep 15 08:05:03.929001 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:03.928950    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-75bcc56485-7tvtl_e886b953-6c7c-4127-8569-cf75a55ecba8/authorino/0.log"
Sep 15 08:05:03.960812 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:03.960760    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-cx79r_c48d52a2-e97c-4db6-b960-c48cf826201c/manager/0.log"
Sep 15 08:05:03.984855 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:03.984817    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-r9tjx_08aeeb1f-4086-4b9e-84df-61f6d93a3534/manager/0.log"
Sep 15 08:05:04.058633 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:04.058593    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-qgxpl_e307cef6-bbf3-496a-a8b7-58ebba9cfe87/manager/0.log"
Sep 15 08:05:04.090230 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:04.090195    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-26rq6_7ac85e8e-0767-4b03-b51f-5f536c71d6dd/limitador/0.log"
Sep 15 08:05:05.904940 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:05.904857    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287fs_a896d384-9995-4ef8-94c3-ca53ab0864e7/node-exporter/0.log"
Sep 15 08:05:05.938641 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:05.937860    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287fs_a896d384-9995-4ef8-94c3-ca53ab0864e7/kube-rbac-proxy/0.log"
Sep 15 08:05:05.960455 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:05.960426    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-287fs_a896d384-9995-4ef8-94c3-ca53ab0864e7/init-textfile/0.log"
Sep 15 08:05:08.250257 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.250217    2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"]
Sep 15 08:05:08.255057 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.255026    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.263010 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.262986    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"]
Sep 15 08:05:08.336751 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.336710    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-proc\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.337039 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.337017    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-sys\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.337289 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.337259    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-lib-modules\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.337406 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.337323    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-podres\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.337478 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.337432    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq4sz\" (UniqueName: \"kubernetes.io/projected/15eea9fb-a801-42b2-b2a4-013dab830e1f-kube-api-access-tq4sz\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.439081 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439035    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-proc\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.439356 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439325    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-proc\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.439552 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439360    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-sys\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.439727 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439698    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-lib-modules\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.440079 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.440057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-podres\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.440204 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439428    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-sys\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.440204 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.439977    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-lib-modules\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.440204 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.440166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq4sz\" (UniqueName: \"kubernetes.io/projected/15eea9fb-a801-42b2-b2a4-013dab830e1f-kube-api-access-tq4sz\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.440352 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.440253    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/15eea9fb-a801-42b2-b2a4-013dab830e1f-podres\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.448369 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.448334    2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq4sz\" (UniqueName: \"kubernetes.io/projected/15eea9fb-a801-42b2-b2a4-013dab830e1f-kube-api-access-tq4sz\") pod \"perf-node-gather-daemonset-rz5xt\" (UID: \"15eea9fb-a801-42b2-b2a4-013dab830e1f\") " pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.569854 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.569773    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:08.883931 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:08.882040    2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"]
Sep 15 08:05:09.122913 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:09.122883    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5778f4bdf6-pmv7t_75b9657e-4de1-4ec1-8bdd-5b849d270762/console/0.log"
Sep 15 08:05:09.665152 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:09.665078    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt" event={"ID":"15eea9fb-a801-42b2-b2a4-013dab830e1f","Type":"ContainerStarted","Data":"e6d2b28db648eb0e94a727d1ea2e16d1a2f7d686e9f2e084f98639b7947366b5"}
Sep 15 08:05:09.665637 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:09.665185    2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt" event={"ID":"15eea9fb-a801-42b2-b2a4-013dab830e1f","Type":"ContainerStarted","Data":"f6ae4f9185fc832733bd4385d060069469718d7f20eda0eaa2d8a1c0bc385f22"}
Sep 15 08:05:09.665637 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:09.665373    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:09.680951 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:09.680893    2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt" podStartSLOduration=1.680874056 podStartE2EDuration="1.680874056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 08:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 08:05:09.679887245 +0000 UTC m=+2797.774053510" watchObservedRunningTime="2026-09-15 08:05:09.680874056 +0000 UTC m=+2797.775040319"
Sep 15 08:05:10.934279 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:10.934250    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dt7v6_7ca0b761-ffd4-409c-bcbd-379522595180/dns/0.log"
Sep 15 08:05:10.953702 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:10.953676    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dt7v6_7ca0b761-ffd4-409c-bcbd-379522595180/kube-rbac-proxy/0.log"
Sep 15 08:05:11.021768 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:11.021741    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-84rbg_912a1a00-fcb7-488e-922c-90863e26694b/dns-node-resolver/0.log"
Sep 15 08:05:11.581600 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:11.581565    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-cd4bbcf87-tkcl4_407f8dee-2679-4257-a1ea-46f81a37ad93/registry/0.log"
Sep 15 08:05:11.625949 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:11.625923    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l5q52_16ce4021-b6c1-472c-b4fe-379a5d2bf4a6/node-ca/0.log"
Sep 15 08:05:13.452555 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:13.452524    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-v8gdn_3536fa4f-d8f7-41e4-b371-a0687e74f611/kube-auth-proxy/0.log"
Sep 15 08:05:13.478727 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:13.478698    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/0.log"
Sep 15 08:05:13.482197 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:13.482166    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/1.log"
Sep 15 08:05:14.472181 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:14.472137    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6vjvf_964bc816-53aa-4e24-bc1c-af2d4dfdf27a/serve-healthcheck-canary/0.log"
Sep 15 08:05:15.044253 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:15.044223    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkw9s_da409ae3-c129-48f5-818c-5b132db016f9/kube-rbac-proxy/0.log"
Sep 15 08:05:15.074265 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:15.074234    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkw9s_da409ae3-c129-48f5-818c-5b132db016f9/exporter/0.log"
Sep 15 08:05:15.103718 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:15.103690    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gkw9s_da409ae3-c129-48f5-818c-5b132db016f9/extractor/0.log"
Sep 15 08:05:15.684169 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:15.684130    2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c8zpm/perf-node-gather-daemonset-rz5xt"
Sep 15 08:05:17.462811 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:17.462775    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-smrhn_cd7c9f17-47bc-4550-a393-c5aaa46943ee/manager/0.log"
Sep 15 08:05:17.823323 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:17.823292    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-q29mg_54dbf0d5-437a-4355-8e8d-c2915d3aaa59/manager/0.log"
Sep 15 08:05:17.884663 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:17.884441    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-656b58fbfb-8bwsq_94ec87e2-035a-473b-a6e9-5da8c4c349f9/manager/0.log"
Sep 15 08:05:18.020888 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:18.020848    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bkj9r_5e58c92a-eecb-4909-bf1b-b00b4059e4b9/manager/0.log"
Sep 15 08:05:18.256248 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:18.256195    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-tsjj8_a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520/manager/0.log"
Sep 15 08:05:18.277781 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:18.277749    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-565d545f7b-tsjj8_a2b9a9d3-0bb7-4f86-ac16-8ca61b69c520/copy-manifests/0.log"
Sep 15 08:05:19.496322 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:19.496290    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-lknc8_347dc5f0-3296-4fc9-85bb-906a75d4e6cf/openshift-lws-operator/0.log"
Sep 15 08:05:19.545740 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:19.545666    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-v8gdn_3536fa4f-d8f7-41e4-b371-a0687e74f611/kube-auth-proxy/0.log"
Sep 15 08:05:19.570815 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:19.570671    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/0.log"
Sep 15 08:05:19.573045 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:19.573014    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/1.log"
Sep 15 08:05:21.918579 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:21.918543    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-77b7bc76bc-v8gdn_3536fa4f-d8f7-41e4-b371-a0687e74f611/kube-auth-proxy/0.log"
Sep 15 08:05:21.937833 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:21.937806    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/0.log"
Sep 15 08:05:21.941478 ip-10-0-141-227 kubenswrapper[2574]: I0915 08:05:21.941452    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-nj4wg_77088409-0655-4a30-a63f-1a7bc5ef1a8b/istio-proxy/1.log"