Sep 17 15:27:58.361540 ip-10-0-141-17 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 15:27:58.361551 ip-10-0-141-17 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 15:27:58.361560 ip-10-0-141-17 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 15:27:58.361763 ip-10-0-141-17 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 15:28:08.442194 ip-10-0-141-17 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 15:28:08.442211 ip-10-0-141-17 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a043e23064564277ba78342440b496d2 --
Sep 17 15:30:06.156784 ip-10-0-141-17 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:30:06.527863 ip-10-0-141-17 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:30:06.527863 ip-10-0-141-17 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:30:06.527863 ip-10-0-141-17 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:30:06.527863 ip-10-0-141-17 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:30:06.527863 ip-10-0-141-17 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:30:06.528563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.527938    2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:30:06.532557 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532531    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.532557 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532556    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532562    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532574    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532578    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532583    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532588    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532592    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532596    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532601    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532605    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532609    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532614    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532618    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532622    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532626    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532641    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532645    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532649    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532654    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532658    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.532697 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532662    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532667    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532671    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532676    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532680    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532684    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532688    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532697    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532701    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532705    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532710    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532714    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532718    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532723    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532727    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532731    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532735    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532740    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532743    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.533446 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532745    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532753    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532757    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532760    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532764    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532767    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532770    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532775    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532778    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532781    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.532784    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533411    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533421    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533426    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533429    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533432    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533434    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533437    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533440    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.533915 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533443    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533447    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533451    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533454    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533457    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533459    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533462    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533465    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533468    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533473    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533476    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533479    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533482    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533485    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533502    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533506    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533509    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533512    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533515    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.534381 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533518    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533522    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533526    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533530    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533535    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533539    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533542    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533545    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533547    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533550    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533553    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533555    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533558    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533561    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533564    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533567    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533570    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533572    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533575    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.534890 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533577    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.533580    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534330    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534336    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534340    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534343    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534346    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534349    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534352    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534355    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534358    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534361    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534364    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534366    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534369    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534372    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534374    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534377    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534382    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534386    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.535355 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534388    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534392    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534395    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534397    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534400    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534402    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534405    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534408    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534410    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534413    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534416    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534419    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534421    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534424    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534426    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534429    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534432    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534435    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534437    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534440    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.535871 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534442    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534445    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534447    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534450    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534452    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534455    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534457    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534460    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534462    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534465    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534469    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534477    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534481    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534484    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534488    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534508    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534513    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534516    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534519    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.536373 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534523    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534526    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534529    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534532    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534535    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534537    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534540    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534542    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534545    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534547    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534550    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534552    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534555    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534558    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534561    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534563    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534566    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534568    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534571    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534574    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.536865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534576    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534579    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534582    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534585    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534587    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534590    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534593    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534596    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534599    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534601    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534604    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534607    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534609    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534612    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534614    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534617    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534620    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534624    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534627    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.537356 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534630    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534633    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.534636    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535008    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535013    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535017    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535020    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535023    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535026    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535028    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535031    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535034    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535038    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535041    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535045    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535048    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535051    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535054    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535057    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.537865 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535061    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535063    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535066    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535068    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535071    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535074    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535076    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535079    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535082    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535084    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535087    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535089    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535092    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535094    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535097    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535099    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535102    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535104    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535107    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535110    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.538341 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535113    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535115    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535118    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535120    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535123    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535126    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535128    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535131    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535133    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535137    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535139    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535142    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535145    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535148    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535151    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535153    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535156    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535159    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535161    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535164    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.538835 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535166    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535169    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535171    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535174    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535176    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535179    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535181    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535184    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535186    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535188    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535191    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535193    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535196    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535198    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535201    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535204    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535206    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535210    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535213    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535216    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.539347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535219    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535222    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535225    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535228    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535230    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535233    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535237    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535240    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535242    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535245    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535248    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535250    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535253    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535255    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535258    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535260    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535263    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535266    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535268    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535271    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.539934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535273    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535276    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.535278    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535352    2564 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535391    2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535398    2564 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535403    2564 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535407    2564 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535410    2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535414    2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535419    2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535422    2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535425    2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535431    2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535435    2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535438    2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535441    2564 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535444    2564 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535447    2564 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535450    2564 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535453    2564 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535457    2564 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535460    2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:30:06.540428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535463    2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535466    2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535470    2564 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535473    2564 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535476    2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535479    2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535483    2564 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535485    2564 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535488    2564 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535505    2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535508    2564 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535511    2564 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535515    2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535518    2564 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535521    2564 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535524    2564 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535526    2564 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535529    2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535534    2564 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535537    2564 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535540    2564 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535543    2564 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535548    2564 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535552    2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:30:06.540995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535555    2564 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535558    2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535561    2564 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535564    2564 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535567    2564 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535570    2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535573    2564 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535576    2564 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535579    2564 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535582    2564 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535585    2564 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535588    2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535591    2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535594    2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535597    2564 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535600    2564 flags.go:64] FLAG: --help="false"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535603    2564 flags.go:64] FLAG: --hostname-override="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535606    2564 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535609    2564 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535612    2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535616    2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535619    2564 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535622    2564 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:30:06.541589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535625    2564 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535627    2564 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535630    2564 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535633    2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535636    2564 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535639    2564 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535642    2564 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535647    2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535650    2564 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535653    2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535656    2564 flags.go:64] FLAG: --lock-file=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535659    2564 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535661    2564 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535664    2564 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535670    2564 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535673    2564 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535676    2564 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535679    2564 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535682    2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535685    2564 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535689    2564 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535691    2564 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535696    2564 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535699    2564 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535703    2564 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:30:06.542178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535706    2564 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535708    2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535711    2564 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535714    2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535717    2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535720    2564 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535723    2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535730    2564 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535733    2564 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535736    2564 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535739    2564 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535742    2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535747    2564 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535750    2564 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535754    2564 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535758    2564 flags.go:64] FLAG: --port="10250"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535761    2564 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535764    2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d39a7b74e2654ff"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535767    2564 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535770    2564 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535773    2564 flags.go:64] FLAG: --register-node="true"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535776    2564 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535780    2564 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535783    2564 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:30:06.542858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535786    2564 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535789    2564 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535793    2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535796    2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535799    2564 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535802    2564 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535805    2564 flags.go:64] FLAG: --runonce="false"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535808    2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535811    2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535813    2564 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535816    2564 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535819    2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535822    2564 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535825    2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535828    2564 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535831    2564 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535833    2564 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535836    2564 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535839    2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535842    2564 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535845    2564 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535847    2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535854    2564 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535858    2564 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535861    2564 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:30:06.543480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535865    2564 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535868    2564 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535871    2564 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535874    2564 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535877    2564 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535880    2564 flags.go:64] FLAG: --v="2"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535884    2564 flags.go:64] FLAG: --version="false"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535889    2564 flags.go:64] FLAG: --vmodule=""
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535893    2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.535896    2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536018    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536022    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536026    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536029    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536032    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536034    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536037    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536039    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536042    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536045    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536048    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536050    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.544087 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536053    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536056    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536058    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536061    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536064    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536066    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536069    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536073    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536077    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536080    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536082    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536087    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536090    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536094    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536097    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536100    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536103    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536106    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536110    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.544642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536113    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536115    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536118    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536121    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536123    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536126    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536129    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536132    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536135    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536137    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536140    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536143    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536145    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536148    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536151    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536153    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536156    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536159    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536161    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536164    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.545117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536168    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536172    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536174    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536177    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536180    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536183    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536185    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536188    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536191    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536194    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536196    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536199    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536204    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536207    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536210    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536213    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536217    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536220    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536222    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.545642 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536225    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536228    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536231    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536234    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536236    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536239    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536242    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536244    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536247    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536250    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536252    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536255    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536257    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536261    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536265    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536267    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536270    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536273    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536275    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.546112 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536278    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536280    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536283    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536286    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536288    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536291    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536293    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536296    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536299    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.536302    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.536772    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.543725    2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.543743    2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.544441    2564 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:30:06.546637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.544450    2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544527    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544535    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544538    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544543    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544547    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544551    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544554    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544556    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544559    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544562    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544565    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544568    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544570    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544573    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544576    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544579    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544581    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544584    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.546998 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544586    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544589    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544592    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544595    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544597    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544600    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544603    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544605    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544608    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544611    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544613    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544617    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544620    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544623    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544625    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544628    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544631    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544633    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544636    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544638    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.547514 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544641    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544644    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544646    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544649    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544652    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544654    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544658    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544662    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544665    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544667    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544670    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544673    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544676    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544679    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544683    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544686    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544688    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544691    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544694    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.548033 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544697    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544699    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544702    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544704    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544707    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544710    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544713    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544715    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544718    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544720    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544723    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544725    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544728    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544730    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544733    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544735    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544738    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544740    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544743    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544745    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.548599 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544748    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544751    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544753    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544756    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544759    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544761    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544764    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544767    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544769    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544772    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544774    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544777    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544779    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544782    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544784    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544787    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544789    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544792    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544796    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544798    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.549088 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544801    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544804    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.544809    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544920    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544924    2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544927    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544930    2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544933    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544935    2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544938    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544940    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544943    2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544946    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544949    2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544951    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:30:06.549616 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544953    2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544956    2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544959    2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544962    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544965    2564 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544967    2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544970    2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544972    2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544975    2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544977    2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544980    2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544982    2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544985    2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544987    2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544990    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544992    2564 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544995    2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.544998    2564 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545000    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545004    2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:30:06.550011 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545007    2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545011    2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545014    2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545017    2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545020    2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545022    2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545025    2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545028    2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545030    2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545033    2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545035    2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545038    2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545041    2564 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545044    2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545046    2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545049    2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545052    2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545054    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545057    2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:30:06.550518 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545059    2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545062    2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545064    2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545067    2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545069    2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545072    2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545075    2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545079    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545082    2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545084    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545088    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545091    2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545094    2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545097    2564 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545100    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545103    2564 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545106    2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545108    2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545111    2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:30:06.550997 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545113    2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545116    2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545119    2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545121    2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545124    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545126    2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545129    2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545134    2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545137    2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545140    2564 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545143    2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545146    2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545149    2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545152    2564 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545155    2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545158    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545161    2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545164    2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545166    2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:30:06.551452 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545169    2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545171    2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545174    2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545176    2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545179    2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545182    2564 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545184    2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545187    2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545189    2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:06.545192    2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.545197    2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.545331    2564 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.547165    2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.547846    2564 server.go:1019] "Starting client certificate rotation"
Sep 17 15:30:06.551934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.547939    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:30:06.552312 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.547987    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:30:06.552312 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.549965    2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:30:06.554243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.554225    2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:30:06.565642 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.565618    2564 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:30:06.568441 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.568426    2564 log.go:25] "Validated CRI v1 image API"
Sep 17 15:30:06.570085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.570069    2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:30:06.573791 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.573763    2564 fs.go:135] Filesystem UUIDs: map[5549d237-a3a6-4114-91f8-302af2e3b833:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 926e9697-07f4-4c2c-b872-736e92ae1df2:/dev/nvme0n1p4]
Sep 17 15:30:06.573864 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.573790    2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:30:06.574684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.574668    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:30:06.579088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.578976    2564 manager.go:216] Machine: {Timestamp:2026-09-17 15:30:06.577754607 +0000 UTC m=+0.307053346 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093323 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269603002d3589a51856d94d69b9c9 SystemUUID:ec269603-002d-3589-a518-56d94d69b9c9 BootID:a043e230-6456-4277-ba78-342440b496d2 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:7d:16:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:7d:16:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4a:79:2c:dc:88:62 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:30:06.579088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.579086    2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:30:06.579194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.579170    2564 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:30:06.582233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.582206    2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:30:06.582417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.582236    2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-17.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:30:06.583510 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.583488    2564 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:30:06.583549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.583512    2564 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:30:06.583549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.583520    2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:30:06.584082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.584073    2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:30:06.585132 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.585121    2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:30:06.589812 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589799    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:30:06.589931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589922    2564 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:30:06.589962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589954    2564 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:30:06.589989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589963    2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:30:06.589989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589977    2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:30:06.589989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589987    2564 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:30:06.590080 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.589995    2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:30:06.590364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.590353    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:30:06.590406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.590371    2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:30:06.592444 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.592426    2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:30:06.595063 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.595044    2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:30:06.595157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.595084    2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:30:06.597413 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597397    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597419    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597427    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597432    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597441    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597447    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597462    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597468    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597477    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:30:06.597484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597484    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:30:06.597743 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597508    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:30:06.597743 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.597516    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:30:06.598206 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.598176    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-17.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:30:06.598300 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.598278    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:30:06.598344 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.598299    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:30:06.598344 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.598312    2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:30:06.599201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.599182    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hfh6z"
Sep 17 15:30:06.601899 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.601883    2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-17.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:30:06.602086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.602074    2564 server.go:1268] "Started kubelet"
Sep 17 15:30:06.602245 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.602182    2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:30:06.604590 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.604542    2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:30:06.604678 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.604632    2564 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:30:06.605066 ip-10-0-141-17 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:30:06.606041 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.605866    2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:30:06.606617 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.606263    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hfh6z"
Sep 17 15:30:06.609244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.608802    2564 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:30:06.610588 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.609645    2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-17.ec2.internal.18d625626d97dbac  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-17.ec2.internal,UID:ip-10-0-141-17.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-17.ec2.internal,},FirstTimestamp:2026-09-17 15:30:06.602042284 +0000 UTC m=+0.331341017,LastTimestamp:2026-09-17 15:30:06.602042284 +0000 UTC m=+0.331341017,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-17.ec2.internal,}"
Sep 17 15:30:06.611445 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.611426    2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:30:06.615947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.615931    2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:30:06.616028 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.615953    2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:30:06.616092 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.616023    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:06.616862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.616653    2564 factory.go:55] Registering systemd factory
Sep 17 15:30:06.616948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.616941    2564 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.616677    2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.616686    2564 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.616978    2564 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617080    2564 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617089    2564 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617176    2564 factory.go:153] Registering CRI-O factory
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617187    2564 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617238    2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617254    2564 factory.go:103] Registering Raw factory
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617269    2564 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:30:06.618032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.617683    2564 manager.go:349] Starting recovery of all containers
Sep 17 15:30:06.618718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.618694    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:06.621538 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.621514    2564 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-17.ec2.internal\" not found" node="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.629139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.629120    2564 manager.go:354] Recovery completed
Sep 17 15:30:06.633827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.633813    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.635714 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.635694    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.635799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.635729    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.635799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.635743    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.636422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.636409    2564 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:30:06.636422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.636420    2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:30:06.636541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.636436    2564 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:30:06.636541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.636444    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.637232 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.637215    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.637323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.637248    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.637323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.637261    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.638707 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.638694    2564 policy_none.go:49] "None policy: Start"
Sep 17 15:30:06.638764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.638710    2564 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:30:06.638764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.638720    2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:30:06.640722 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.640709    2564 policy_none.go:47] "Start"
Sep 17 15:30:06.674202 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674175    2564 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.674241    2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674256    2564 server.go:87] "Starting device plugin registration server"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674657    2564 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674668    2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674892    2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674985    2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.674993    2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.675364    2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:30:06.696362 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.675403    2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:06.754862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.754824    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:30:06.756102 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.756083    2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:30:06.756201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.756148    2564 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:30:06.756201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.756178    2564 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:30:06.756284 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.756223    2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:30:06.758563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.758535    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:06.775343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.775317    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.776342 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.776314    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.776342 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.776346    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.776591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.776356    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.776591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.776383    2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.784075 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.783982    2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.784075 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.784012    2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-17.ec2.internal\": node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:06.802145 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.802113    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:06.856580 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.856535    2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"]
Sep 17 15:30:06.856717 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.856643    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.858230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.858208    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.858316 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.858236    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.858316 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.858248    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.860318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.860302    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.860444 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.860426    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.860479 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.860467    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.861092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861074    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.861189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861091    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.861189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861117    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.861189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861129    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.861189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861099    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.861189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.861178    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.863152 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.863136    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.863217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.863165    2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:30:06.863957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.863939    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:30:06.864030 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.863979    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:30:06.864030 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.863995    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:30:06.884514 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.884472    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-17.ec2.internal\" not found" node="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.888913 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.888893    2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-17.ec2.internal\" not found" node="ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.902758 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:06.902731    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:06.918552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.918519    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.918552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.918552    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27dcf26562be69b4ba8f5d6ac22207fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-17.ec2.internal\" (UID: \"27dcf26562be69b4ba8f5d6ac22207fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:06.918752 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:06.918568    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.003427 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:07.003397    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:07.018756 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.018822 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018762    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.018822 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27dcf26562be69b4ba8f5d6ac22207fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-17.ec2.internal\" (UID: \"27dcf26562be69b4ba8f5d6ac22207fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.018897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.018897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018852    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc11645305a3708ac42e199a9dd16b8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal\" (UID: \"9bc11645305a3708ac42e199a9dd16b8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.018897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.018881    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/27dcf26562be69b4ba8f5d6ac22207fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-17.ec2.internal\" (UID: \"27dcf26562be69b4ba8f5d6ac22207fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.104269 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:07.104189    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:07.186809 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.186771    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.191529 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:07.191506    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:07.204551 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:07.204523    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.067253 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:09.067223    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.167738 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:09.167677    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.268320 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:09.268298    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.309829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.309806    2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:30:09.310201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.309931    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:30:09.310201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.309951    2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:30:09.369077 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:09.369055    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.375386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.375361    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:25:08 +0000 UTC" deadline="2028-04-09 10:29:42.890489581 +0000 UTC"
Sep 17 15:30:09.375386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.375385    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13674h59m33.515107152s"
Sep 17 15:30:09.469668 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:09.469640    2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-17.ec2.internal\" not found"
Sep 17 15:30:09.491719 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.491693    2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:09.538077 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:09.538050    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27dcf26562be69b4ba8f5d6ac22207fe.slice/crio-6c57847e5d398d2cc71e75de2b4efef86b5706cd773b5016365a443ac4c7ca57 WatchSource:0}: Error finding container 6c57847e5d398d2cc71e75de2b4efef86b5706cd773b5016365a443ac4c7ca57: Status 404 returned error can't find the container with id 6c57847e5d398d2cc71e75de2b4efef86b5706cd773b5016365a443ac4c7ca57
Sep 17 15:30:09.542151 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.542133    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:30:09.578389 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.578372    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:09.591759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.591744    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:30:09.593216 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.593204    2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal"
Sep 17 15:30:09.600001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.599986    2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:30:09.785580 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:09.785558    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc11645305a3708ac42e199a9dd16b8.slice/crio-44fa75ef5f124deab1c11e138e03a1c43b03898d0763cab300046503e0daca1b WatchSource:0}: Error finding container 44fa75ef5f124deab1c11e138e03a1c43b03898d0763cab300046503e0daca1b: Status 404 returned error can't find the container with id 44fa75ef5f124deab1c11e138e03a1c43b03898d0763cab300046503e0daca1b
Sep 17 15:30:09.810763 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:09.810741    2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:10.150337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.150267    2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:10.353381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.353349    2564 apiserver.go:52] "Watching apiserver"
Sep 17 15:30:10.360717 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.360686    2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:30:10.361532 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.361501    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal","openshift-dns/node-resolver-6r2g7","openshift-image-registry/node-ca-cjwxc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal","openshift-multus/network-metrics-daemon-8m8nk","kube-system/konnectivity-agent-vbfj6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft","openshift-cluster-node-tuning-operator/tuned-6kn45","openshift-multus/multus-additional-cni-plugins-c99tn","openshift-multus/multus-vkkkv","openshift-network-diagnostics/network-check-target-67sjn","openshift-network-operator/iptables-alerter-5pvpd","openshift-ovn-kubernetes/ovnkube-node-jp8g7","kube-system/global-pull-secret-syncer-kg4qt"]
Sep 17 15:30:10.364398 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.364375    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.366661 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.366643    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.367229 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.367208    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:30:10.367652 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.367636    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7fpxx\""
Sep 17 15:30:10.367752 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.367653    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.368163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.368146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:30:10.368544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.368525    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.368915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.368803    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:30:10.368915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.368855    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.369585 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.369563    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.369810 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.369794    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:30:10.370304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.370286    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gz2nh\""
Sep 17 15:30:10.370382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.370302    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:30:10.371040 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371013    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.371124 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371108    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:30:10.371181 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371162    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.371252 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371234    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.371322 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.371299    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:10.371402 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371385    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:30:10.371451 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.371401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.373726 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.373708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.373795 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.373770    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:10.375673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.375656    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dclrc\""
Sep 17 15:30:10.375897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.375885    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.377913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.377896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8dwgb\""
Sep 17 15:30:10.378102 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.378084    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:30:10.378223 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.378206    2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:30:10.378320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.378308    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.378757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.378735    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:30:10.381306 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.381276    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.381563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.381436    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:30:10.381563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.381553    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:30:10.381895 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.381876    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f75s5\""
Sep 17 15:30:10.383739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.383644    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.384170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.383833    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-q562k\""
Sep 17 15:30:10.385551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.385535    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.385750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.385738    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.385871 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.385853    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:30:10.386132 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.386117    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.386397 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.386374    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.386498 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.386381    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.386912 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.386892    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-j9xbm\""
Sep 17 15:30:10.388211 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388194    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-slash\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388222    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-netns\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-registration-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.388296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.388414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388307    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-sys-fs\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.388414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388358    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-socket-dir-parent\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.388414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388388    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-netns\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.388544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388413    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-etc-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388436    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-ovn\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388458    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b4272a0-bb05-4cb8-879e-afacd2edb750-agent-certs\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.388544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-node-log\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388544 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388503    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-script-lib\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388817 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rhr\" (UniqueName: \"kubernetes.io/projected/7daf6886-a7af-4b98-834f-5dee8153323a-kube-api-access-w9rhr\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.388817 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388812    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-system-cni-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.388907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388828    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.388907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-systemd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.388907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388870    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.389037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388904    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.389037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7daf6886-a7af-4b98-834f-5dee8153323a-serviceca\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.389037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.388965    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-binary-copy\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.389037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389022    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389059    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-config\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbv2b\" (UniqueName: \"kubernetes.io/projected/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-kube-api-access-bbv2b\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389124    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-systemd-units\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-bin\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389181    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.389214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389207    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.389475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389231    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-conf-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.389475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389252    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nmq4\" (UniqueName: \"kubernetes.io/projected/34369b40-593d-40a5-a244-cbcb4d0cea24-kube-api-access-9nmq4\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.389475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.389774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.389754    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:10.389922 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.389892    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:10.390185 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390142    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b4272a0-bb05-4cb8-879e-afacd2edb750-konnectivity-ca\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.390262 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-var-lib-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.390262 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390221    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-device-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.390262 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390237    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n48m4\""
Sep 17 15:30:10.390262 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390246    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-system-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390270    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-bin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390292    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-netd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390334    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02bc3c35-2376-4376-a493-ad3f3b677faa-ovn-node-metrics-cert\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390374    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-kubelet\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390401    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.390456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390404    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390461    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-kubelet-config\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390498    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-dbus\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4vv5\" (UniqueName: \"kubernetes.io/projected/276aa7f1-1096-4e27-bb04-9e20d484d804-kube-api-access-f4vv5\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390586    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-k8s-cni-cncf-io\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-multus\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-multus-certs\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390675    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-etc-kubernetes\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.390710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390695    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-env-overrides\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390745    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-cnibin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390809    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-hostroot\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkkl\" (UniqueName: \"kubernetes.io/projected/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-kube-api-access-svkkl\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390876    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-kubelet\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390903    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-log-socket\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390925    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hgtw\" (UniqueName: \"kubernetes.io/projected/02bc3c35-2376-4376-a493-ad3f3b677faa-kube-api-access-2hgtw\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390955    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-etc-selinux\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.390984    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-daemon-config\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.391060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.391028    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7daf6886-a7af-4b98-834f-5dee8153323a-host\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.391434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.391076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cnibin\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.391434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.391103    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-socket-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.391659 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.391531    2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:30:10.392164 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.391149    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.392239 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.392189    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-cni-binary-copy\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.392288 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.392254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.395564 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.395544    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.397743 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.397720    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:30:10.397833 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.397727    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sncnc\""
Sep 17 15:30:10.398183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.398153    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:30:10.398783 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.398763    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:30:10.418749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.418732    2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-rbbft"
Sep 17 15:30:10.429961 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.429936    2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-rbbft"
Sep 17 15:30:10.480189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.480169    2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:30:10.492509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492487    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-config\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492548    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbv2b\" (UniqueName: \"kubernetes.io/projected/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-kube-api-access-bbv2b\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492569    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-conf\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492593    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-lib-modules\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492630    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-systemd-units\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492639    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-bin\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.492710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492703    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-systemd-units\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-bin\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492784    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.492843    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-conf-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492922    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.492938    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.99291636 +0000 UTC m=+2.960094174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-conf-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.492988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-kubernetes\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493025    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-sys\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.493093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhf7w\" (UniqueName: \"kubernetes.io/projected/168b3820-7adb-49bc-a51a-aa4a8db59bcc-kube-api-access-hhf7w\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493110    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nmq4\" (UniqueName: \"kubernetes.io/projected/34369b40-593d-40a5-a244-cbcb4d0cea24-kube-api-access-9nmq4\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493142    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493145    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-config\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b4272a0-bb05-4cb8-879e-afacd2edb750-konnectivity-ca\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.493241    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493299    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-var-lib-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.493317    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:10.99329879 +0000 UTC m=+2.960476607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-device-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493351    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-var-lib-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493376    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-system-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-bin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493426    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-netd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493400    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-device-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493450    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02bc3c35-2376-4376-a493-ad3f3b677faa-ovn-node-metrics-cert\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-bin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493474    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-kubelet\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.493553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493484    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-cni-netd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-system-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-var-lib-kubelet\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493567    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-kubelet\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493603    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-tuned\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493670    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc9c39f-56af-494c-b585-cb5f46588bc6-host-slash\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9b4272a0-bb05-4cb8-879e-afacd2edb750-konnectivity-ca\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493697    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493725    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-kubelet-config\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-dbus\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493750    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493790    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-kubelet-config\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493806    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4vv5\" (UniqueName: \"kubernetes.io/projected/276aa7f1-1096-4e27-bb04-9e20d484d804-kube-api-access-f4vv5\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-dbus\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493832    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhw6v\" (UniqueName: \"kubernetes.io/projected/0f588baf-81cd-4efb-a767-bdbd544e38d4-kube-api-access-jhw6v\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493858    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493882    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-k8s-cni-cncf-io\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.494355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493914    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-multus\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493942    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-multus-certs\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493948    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-k8s-cni-cncf-io\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.493964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-etc-kubernetes\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494003    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-env-overrides\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-multus-certs\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-var-lib-cni-multus\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-cnibin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494050    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-etc-kubernetes\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-hostroot\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494087    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-hostroot\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkkl\" (UniqueName: \"kubernetes.io/projected/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-kube-api-access-svkkl\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494117    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-kubelet\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494133    2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494176    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-log-socket\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-cnibin\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-log-socket\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494217    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hgtw\" (UniqueName: \"kubernetes.io/projected/02bc3c35-2376-4376-a493-ad3f3b677faa-kube-api-access-2hgtw\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.495194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494241    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-etc-selinux\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494247    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-kubelet\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494269    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-daemon-config\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-etc-selinux\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494507    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7daf6886-a7af-4b98-834f-5dee8153323a-host\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494433    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-env-overrides\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494551    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7daf6886-a7af-4b98-834f-5dee8153323a-host\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494541    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cnibin\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494571    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cnibin\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494607    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-run\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-socket-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494682    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494707    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-cni-binary-copy\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494749    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-daemon-config\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494755    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-tmp\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494779    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-slash\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494791    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-socket-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494801    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-cni-dir\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494803    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-netns\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-run-netns\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-slash\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494840    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-registration-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494922    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-sys-fs\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494919    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-registration-dir\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494938    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-socket-dir-parent\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494956    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-netns\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494975    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f588baf-81cd-4efb-a767-bdbd544e38d4-hosts-file\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.494984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/276aa7f1-1096-4e27-bb04-9e20d484d804-sys-fs\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-host-run-netns\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495018    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-systemd\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495026    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/34369b40-593d-40a5-a244-cbcb4d0cea24-multus-socket-dir-parent\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495046    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc9c39f-56af-494c-b585-cb5f46588bc6-iptables-alerter-script\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495074    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt45q\" (UniqueName: \"kubernetes.io/projected/9bc9c39f-56af-494c-b585-cb5f46588bc6-kube-api-access-rt45q\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.496884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495099    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f588baf-81cd-4efb-a767-bdbd544e38d4-tmp-dir\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495125    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-etc-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495150    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-ovn\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-etc-openvswitch\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495174    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b4272a0-bb05-4cb8-879e-afacd2edb750-agent-certs\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495196    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-host\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-ovn\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495221    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-node-log\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495245    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-node-log\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/34369b40-593d-40a5-a244-cbcb4d0cea24-cni-binary-copy\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495270    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-script-lib\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rhr\" (UniqueName: \"kubernetes.io/projected/7daf6886-a7af-4b98-834f-5dee8153323a-kube-api-access-w9rhr\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-system-cni-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-systemd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495393    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-run-systemd\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495394    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-system-cni-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.497689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495473    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495496    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495517    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02bc3c35-2376-4376-a493-ad3f3b677faa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495520    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-modprobe-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysconfig\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495588    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7daf6886-a7af-4b98-834f-5dee8153323a-serviceca\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495626    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-binary-copy\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495651    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/02bc3c35-2376-4376-a493-ad3f3b677faa-ovnkube-script-lib\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.495998    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.496044    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7daf6886-a7af-4b98-834f-5dee8153323a-serviceca\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.496115    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-cni-binary-copy\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.498487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.496165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.499053 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.498650    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/02bc3c35-2376-4376-a493-ad3f3b677faa-ovn-node-metrics-cert\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.499053 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.498757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9b4272a0-bb05-4cb8-879e-afacd2edb750-agent-certs\") pod \"konnectivity-agent-vbfj6\" (UID: \"9b4272a0-bb05-4cb8-879e-afacd2edb750\") " pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.509147 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.509122    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbv2b\" (UniqueName: \"kubernetes.io/projected/aab0eb57-6e0b-4891-8aa6-f20a9a5ec930-kube-api-access-bbv2b\") pod \"multus-additional-cni-plugins-c99tn\" (UID: \"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930\") " pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.509248 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.509124    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nmq4\" (UniqueName: \"kubernetes.io/projected/34369b40-593d-40a5-a244-cbcb4d0cea24-kube-api-access-9nmq4\") pod \"multus-vkkkv\" (UID: \"34369b40-593d-40a5-a244-cbcb4d0cea24\") " pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.509467 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.509442    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rhr\" (UniqueName: \"kubernetes.io/projected/7daf6886-a7af-4b98-834f-5dee8153323a-kube-api-access-w9rhr\") pod \"node-ca-cjwxc\" (UID: \"7daf6886-a7af-4b98-834f-5dee8153323a\") " pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.509984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.509963    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkkl\" (UniqueName: \"kubernetes.io/projected/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-kube-api-access-svkkl\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:10.510177 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.510147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hgtw\" (UniqueName: \"kubernetes.io/projected/02bc3c35-2376-4376-a493-ad3f3b677faa-kube-api-access-2hgtw\") pod \"ovnkube-node-jp8g7\" (UID: \"02bc3c35-2376-4376-a493-ad3f3b677faa\") " pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.510495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.510477    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4vv5\" (UniqueName: \"kubernetes.io/projected/276aa7f1-1096-4e27-bb04-9e20d484d804-kube-api-access-f4vv5\") pod \"aws-ebs-csi-driver-node-xrhft\" (UID: \"276aa7f1-1096-4e27-bb04-9e20d484d804\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.525293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.525226    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal" event={"ID":"9bc11645305a3708ac42e199a9dd16b8","Type":"ContainerStarted","Data":"44fa75ef5f124deab1c11e138e03a1c43b03898d0763cab300046503e0daca1b"}
Sep 17 15:30:10.526665 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.526605    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal" event={"ID":"27dcf26562be69b4ba8f5d6ac22207fe","Type":"ContainerStarted","Data":"6c57847e5d398d2cc71e75de2b4efef86b5706cd773b5016365a443ac4c7ca57"}
Sep 17 15:30:10.596585 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-tuned\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc9c39f-56af-494c-b585-cb5f46588bc6-host-slash\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.596720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596640    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhw6v\" (UniqueName: \"kubernetes.io/projected/0f588baf-81cd-4efb-a767-bdbd544e38d4-kube-api-access-jhw6v\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.596720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:10.596720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596698    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-run\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596698    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc9c39f-56af-494c-b585-cb5f46588bc6-host-slash\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-tmp\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f588baf-81cd-4efb-a767-bdbd544e38d4-hosts-file\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-systemd\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596798    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-run\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc9c39f-56af-494c-b585-cb5f46588bc6-iptables-alerter-script\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596839    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt45q\" (UniqueName: \"kubernetes.io/projected/9bc9c39f-56af-494c-b585-cb5f46588bc6-kube-api-access-rt45q\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0f588baf-81cd-4efb-a767-bdbd544e38d4-hosts-file\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-systemd\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f588baf-81cd-4efb-a767-bdbd544e38d4-tmp-dir\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596963    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-host\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.596997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.596992    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-modprobe-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysconfig\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597066    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysconfig\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-host\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597102    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-conf\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597153    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-lib-modules\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-modprobe-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597203    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-kubernetes\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f588baf-81cd-4efb-a767-bdbd544e38d4-tmp-dir\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-sys\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597240    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-lib-modules\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597249    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-d\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhf7w\" (UniqueName: \"kubernetes.io/projected/168b3820-7adb-49bc-a51a-aa4a8db59bcc-kube-api-access-hhf7w\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597275    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-sysctl-conf\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597290    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-kubernetes\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597295    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-sys\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597308    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-var-lib-kubelet\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.597500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597376    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/168b3820-7adb-49bc-a51a-aa4a8db59bcc-var-lib-kubelet\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.598418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.597409    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc9c39f-56af-494c-b585-cb5f46588bc6-iptables-alerter-script\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.599369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.599343    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-tmp\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.599523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.599504    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/168b3820-7adb-49bc-a51a-aa4a8db59bcc-etc-tuned\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.603257 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.603237    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:10.603349 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.603264    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:10.603349 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.603294    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:10.603448 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:10.603357    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:11.10334039 +0000 UTC m=+3.070518190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:10.605360 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.605330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhw6v\" (UniqueName: \"kubernetes.io/projected/0f588baf-81cd-4efb-a767-bdbd544e38d4-kube-api-access-jhw6v\") pod \"node-resolver-6r2g7\" (UID: \"0f588baf-81cd-4efb-a767-bdbd544e38d4\") " pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.605545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.605526    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt45q\" (UniqueName: \"kubernetes.io/projected/9bc9c39f-56af-494c-b585-cb5f46588bc6-kube-api-access-rt45q\") pod \"iptables-alerter-5pvpd\" (UID: \"9bc9c39f-56af-494c-b585-cb5f46588bc6\") " pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.606382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.606365    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhf7w\" (UniqueName: \"kubernetes.io/projected/168b3820-7adb-49bc-a51a-aa4a8db59bcc-kube-api-access-hhf7w\") pod \"tuned-6kn45\" (UID: \"168b3820-7adb-49bc-a51a-aa4a8db59bcc\") " pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.675999 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.675946    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vkkkv"
Sep 17 15:30:10.680570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.680180    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:10.688758 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.688741    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cjwxc"
Sep 17 15:30:10.696425 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.696409    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:10.701931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.701915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c99tn"
Sep 17 15:30:10.711539 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.711522    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5pvpd"
Sep 17 15:30:10.718121 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.718105    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft"
Sep 17 15:30:10.723863 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.723849    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6r2g7"
Sep 17 15:30:10.728422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.728408    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kn45"
Sep 17 15:30:10.864047 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:10.864018    2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:30:11.000383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.000306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:11.000383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.000353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:11.000570 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.000462    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:11.000570 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.000485    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:11.000570 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.000530    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:12.000510501 +0000 UTC m=+3.967688322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:11.000570 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.000553    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:12.000541977 +0000 UTC m=+3.967719794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:11.073212 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.073174    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7daf6886_a7af_4b98_834f_5dee8153323a.slice/crio-7ce5e3387954990ef85c04156586eddff947fb0ce4e156da998e370b25a9f8f8 WatchSource:0}: Error finding container 7ce5e3387954990ef85c04156586eddff947fb0ce4e156da998e370b25a9f8f8: Status 404 returned error can't find the container with id 7ce5e3387954990ef85c04156586eddff947fb0ce4e156da998e370b25a9f8f8
Sep 17 15:30:11.079528 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.079503    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc9c39f_56af_494c_b585_cb5f46588bc6.slice/crio-c385e3590bfc93f8ed355723d8ab2e9da0890beb9abc766a97d631dadea722d9 WatchSource:0}: Error finding container c385e3590bfc93f8ed355723d8ab2e9da0890beb9abc766a97d631dadea722d9: Status 404 returned error can't find the container with id c385e3590bfc93f8ed355723d8ab2e9da0890beb9abc766a97d631dadea722d9
Sep 17 15:30:11.080730 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.080635    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f588baf_81cd_4efb_a767_bdbd544e38d4.slice/crio-edc5b418fdcdad20e73ced6de4055b475e3de926985e3be80dd0d915a89bb71f WatchSource:0}: Error finding container edc5b418fdcdad20e73ced6de4055b475e3de926985e3be80dd0d915a89bb71f: Status 404 returned error can't find the container with id edc5b418fdcdad20e73ced6de4055b475e3de926985e3be80dd0d915a89bb71f
Sep 17 15:30:11.081751 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.081539    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276aa7f1_1096_4e27_bb04_9e20d484d804.slice/crio-19f9f7aa9c22bb7b119cec91d1a8b7cf680c35a708bfe78b2368690c2181d3bb WatchSource:0}: Error finding container 19f9f7aa9c22bb7b119cec91d1a8b7cf680c35a708bfe78b2368690c2181d3bb: Status 404 returned error can't find the container with id 19f9f7aa9c22bb7b119cec91d1a8b7cf680c35a708bfe78b2368690c2181d3bb
Sep 17 15:30:11.082975 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.082881    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bc3c35_2376_4376_a493_ad3f3b677faa.slice/crio-1050b1937c74a42ae986b632562ece49a62fb7d0063658112d95cffe31845606 WatchSource:0}: Error finding container 1050b1937c74a42ae986b632562ece49a62fb7d0063658112d95cffe31845606: Status 404 returned error can't find the container with id 1050b1937c74a42ae986b632562ece49a62fb7d0063658112d95cffe31845606
Sep 17 15:30:11.083993 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.083605    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaab0eb57_6e0b_4891_8aa6_f20a9a5ec930.slice/crio-9a5bd3af43bdcd04ea493263cd428f5b837ecb882400c8a4f95e6c6854ee2d3a WatchSource:0}: Error finding container 9a5bd3af43bdcd04ea493263cd428f5b837ecb882400c8a4f95e6c6854ee2d3a: Status 404 returned error can't find the container with id 9a5bd3af43bdcd04ea493263cd428f5b837ecb882400c8a4f95e6c6854ee2d3a
Sep 17 15:30:11.105372 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.105340    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34369b40_593d_40a5_a244_cbcb4d0cea24.slice/crio-ad19d8e63793bb36d7ec8495f7b822b6cf43f827e592efd2d550b6fe4b15ec4d WatchSource:0}: Error finding container ad19d8e63793bb36d7ec8495f7b822b6cf43f827e592efd2d550b6fe4b15ec4d: Status 404 returned error can't find the container with id ad19d8e63793bb36d7ec8495f7b822b6cf43f827e592efd2d550b6fe4b15ec4d
Sep 17 15:30:11.108118 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.108097    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4272a0_bb05_4cb8_879e_afacd2edb750.slice/crio-986d9580637510ed1178fdbf99948985c6843c61182dbd3c722d16b39258e592 WatchSource:0}: Error finding container 986d9580637510ed1178fdbf99948985c6843c61182dbd3c722d16b39258e592: Status 404 returned error can't find the container with id 986d9580637510ed1178fdbf99948985c6843c61182dbd3c722d16b39258e592
Sep 17 15:30:11.108343 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:11.108322    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168b3820_7adb_49bc_a51a_aa4a8db59bcc.slice/crio-6af94aac14174fa0060b988e70024755aa55cda06c65baeea951669bda88d3e1 WatchSource:0}: Error finding container 6af94aac14174fa0060b988e70024755aa55cda06c65baeea951669bda88d3e1: Status 404 returned error can't find the container with id 6af94aac14174fa0060b988e70024755aa55cda06c65baeea951669bda88d3e1
Sep 17 15:30:11.202500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.202478    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:11.202596 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.202584    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:11.202659 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.202602    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:11.202659 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.202625    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:11.202724 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:11.202666    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:12.202653145 +0000 UTC m=+4.169830941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:11.433115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.433039    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:25:10 +0000 UTC" deadline="2028-06-19 11:14:47.68094857 +0000 UTC"
Sep 17 15:30:11.433115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.433073    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15379h44m36.247879732s"
Sep 17 15:30:11.529631 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.529586    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kn45" event={"ID":"168b3820-7adb-49bc-a51a-aa4a8db59bcc","Type":"ContainerStarted","Data":"6af94aac14174fa0060b988e70024755aa55cda06c65baeea951669bda88d3e1"}
Sep 17 15:30:11.530958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.530924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbfj6" event={"ID":"9b4272a0-bb05-4cb8-879e-afacd2edb750","Type":"ContainerStarted","Data":"986d9580637510ed1178fdbf99948985c6843c61182dbd3c722d16b39258e592"}
Sep 17 15:30:11.532354 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.532328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerStarted","Data":"9a5bd3af43bdcd04ea493263cd428f5b837ecb882400c8a4f95e6c6854ee2d3a"}
Sep 17 15:30:11.533769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.533736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"1050b1937c74a42ae986b632562ece49a62fb7d0063658112d95cffe31845606"}
Sep 17 15:30:11.535035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.535008    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6r2g7" event={"ID":"0f588baf-81cd-4efb-a767-bdbd544e38d4","Type":"ContainerStarted","Data":"edc5b418fdcdad20e73ced6de4055b475e3de926985e3be80dd0d915a89bb71f"}
Sep 17 15:30:11.536321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.536294    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pvpd" event={"ID":"9bc9c39f-56af-494c-b585-cb5f46588bc6","Type":"ContainerStarted","Data":"c385e3590bfc93f8ed355723d8ab2e9da0890beb9abc766a97d631dadea722d9"}
Sep 17 15:30:11.538410 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.538373    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal" event={"ID":"27dcf26562be69b4ba8f5d6ac22207fe","Type":"ContainerStarted","Data":"bd2cb9a121e1f5ce90d07aee840f79b8c0b218c44578cf1330b9a7e2ce779db8"}
Sep 17 15:30:11.539735 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.539712    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkkkv" event={"ID":"34369b40-593d-40a5-a244-cbcb4d0cea24","Type":"ContainerStarted","Data":"ad19d8e63793bb36d7ec8495f7b822b6cf43f827e592efd2d550b6fe4b15ec4d"}
Sep 17 15:30:11.541176 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.541147    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft" event={"ID":"276aa7f1-1096-4e27-bb04-9e20d484d804","Type":"ContainerStarted","Data":"19f9f7aa9c22bb7b119cec91d1a8b7cf680c35a708bfe78b2368690c2181d3bb"}
Sep 17 15:30:11.542526 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.542499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cjwxc" event={"ID":"7daf6886-a7af-4b98-834f-5dee8153323a","Type":"ContainerStarted","Data":"7ce5e3387954990ef85c04156586eddff947fb0ce4e156da998e370b25a9f8f8"}
Sep 17 15:30:11.568509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:11.568458    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-17.ec2.internal" podStartSLOduration=2.5684394839999998 podStartE2EDuration="2.568439484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:11.567831797 +0000 UTC m=+3.535009615" watchObservedRunningTime="2026-09-17 15:30:11.568439484 +0000 UTC m=+3.535617297"
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.008889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.008935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.009084    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.009139    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:14.009120263 +0000 UTC m=+5.976298076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.009496    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:12.009562 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.009539    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:14.009525745 +0000 UTC m=+5.976703546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:12.210971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.210889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:12.211193 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.211054    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:12.211193 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.211074    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:12.211193 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.211085    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:12.211193 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.211141    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:14.211122774 +0000 UTC m=+6.178300583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:12.433329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.433231    2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:25:10 +0000 UTC" deadline="2028-04-11 09:33:50.331354224 +0000 UTC"
Sep 17 15:30:12.433329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.433290    2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13722h3m37.898067779s"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.518736    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.518764    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.518877    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.519276    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.519337    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:12.519495 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:12.519413    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:12.550297 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.550267    2564 generic.go:356] "Generic (PLEG): container finished" podID="9bc11645305a3708ac42e199a9dd16b8" containerID="5806576e5c6024a747e532ecafe1d06d50fe448e27d8bf0ff0b80c4b2b4dc268" exitCode=0
Sep 17 15:30:12.551171 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:12.551143    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal" event={"ID":"9bc11645305a3708ac42e199a9dd16b8","Type":"ContainerDied","Data":"5806576e5c6024a747e532ecafe1d06d50fe448e27d8bf0ff0b80c4b2b4dc268"}
Sep 17 15:30:13.563623 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:13.563525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal" event={"ID":"9bc11645305a3708ac42e199a9dd16b8","Type":"ContainerStarted","Data":"345bcf11d3af30018bd5b98a5fd73cb19ccbd6af9bece7337dec74099b7de165"}
Sep 17 15:30:13.580318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:13.580268    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-17.ec2.internal" podStartSLOduration=4.580249522 podStartE2EDuration="4.580249522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:13.578945573 +0000 UTC m=+5.546123393" watchObservedRunningTime="2026-09-17 15:30:13.580249522 +0000 UTC m=+5.547427339"
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.026176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.026228    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.026380    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.026442    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:18.026422785 +0000 UTC m=+9.993600601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.026880    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:14.026958 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.026921    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:18.026910105 +0000 UTC m=+9.994087902 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:14.228198 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.228160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:14.228403 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.228385    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:14.228469 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.228411    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:14.228469 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.228424    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:14.228566 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.228479    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:18.228461775 +0000 UTC m=+10.195639575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:14.523667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.523576    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:14.523820 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.523710    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:14.524098 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.524071    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:14.524202 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.524170    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:14.524300 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:14.524228    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:14.524355 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:14.524325    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:16.519153 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:16.518896    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:16.519563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:16.518963    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:16.519563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:16.518978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:16.519563 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:16.519298    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:16.519563 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:16.519400    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:16.519563 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:16.519496    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:18.062486 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.062447    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:18.062951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.062503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:18.062951 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.062667    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:18.062951 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.062736    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:26.062718629 +0000 UTC m=+18.029896433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:18.062951 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.062667    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:18.062951 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.062834    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:26.062810414 +0000 UTC m=+18.029988224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:18.264167 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.264098    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:18.264327 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.264275    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:18.264327 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.264298    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:18.264327 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.264311    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:18.264491 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.264375    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:26.264357824 +0000 UTC m=+18.231535632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:18.520133 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.520035    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:18.520277 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.520137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:18.520277 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.520150    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:18.520277 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.520246    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:18.520437 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:18.520297    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:18.520437 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:18.520371    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:20.519207 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:20.519169    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:20.519677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:20.519296    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:20.519677 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:20.519308    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:20.519677 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:20.519405    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:20.519677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:20.519438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:20.519677 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:20.519557    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:22.518824 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:22.518772    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:22.519276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:22.518782    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:22.519276 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:22.518887    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:22.519276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:22.518903    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:22.519276 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:22.519024    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:22.519276 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:22.519118    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:24.519520 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:24.519487    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:24.520018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:24.519530    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:24.520018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:24.519475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:24.520018 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:24.519623    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:24.520018 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:24.519734    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:24.520018 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:24.519829    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:26.119979 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.119927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:26.120500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.119991    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:26.120500 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.120097    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:26.120500 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.120114    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:26.120500 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.120185    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.120163574 +0000 UTC m=+34.087341384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:26.120500 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.120208    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.120197562 +0000 UTC m=+34.087375358 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:26.320995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.320958    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:26.321175 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.321131    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:26.321175 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.321156    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:26.321175 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.321166    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:26.321305 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.321223    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:30:42.321207406 +0000 UTC m=+34.288385210 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:26.519085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.518997    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:26.519228 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.519118    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:26.519228 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.519133    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:26.519340 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.519238    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:26.519340 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:26.519298    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:26.519417 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:26.519356    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:28.519650 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:28.519607    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:28.520110 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:28.519672    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:28.520110 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:28.519735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:28.520110 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:28.519779    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:28.520110 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:28.519884    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:28.520110 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:28.519968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:29.596818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.596639    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cjwxc" event={"ID":"7daf6886-a7af-4b98-834f-5dee8153323a","Type":"ContainerStarted","Data":"fbdfd4cdd711ecbba5b7febb4384e987a809a50a3242a44d44fe06f1146d0940"}
Sep 17 15:30:29.598021 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.597999    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kn45" event={"ID":"168b3820-7adb-49bc-a51a-aa4a8db59bcc","Type":"ContainerStarted","Data":"0353f26332e9556859279c0e6fc53574e63ffef5e9394eb0f193261718494496"}
Sep 17 15:30:29.599404 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.599382    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vbfj6" event={"ID":"9b4272a0-bb05-4cb8-879e-afacd2edb750","Type":"ContainerStarted","Data":"7f60088448cbaa7e6f3aca5c9d9c698eaf8b57500d4ac43b7101dcb8a6f0d582"}
Sep 17 15:30:29.600980 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.600961    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="425e22bba6f54b229ae672a5f9247fc9e893027a6f11b4d993a083108cb511c2" exitCode=0
Sep 17 15:30:29.601041 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.601025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"425e22bba6f54b229ae672a5f9247fc9e893027a6f11b4d993a083108cb511c2"}
Sep 17 15:30:29.603691 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.603672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"b30fbea38680a1f39263f671d9f1188954e316d99104d656df73c0b9e83c85a0"}
Sep 17 15:30:29.603771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.603742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"f21bacfc2a9b5fe174f849153d227032df1167044a0cae6c643cd1ec1d04a966"}
Sep 17 15:30:29.603895 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.603810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"fbc676bc1ab74277ac7f43e26b1ff770f634be964835e5e4f0dc27f55be5e9eb"}
Sep 17 15:30:29.603895 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.603858    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"e2e43416d8507a468527717149d990b6561d4a33ddc14235093357e6923297dc"}
Sep 17 15:30:29.605086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.605066    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6r2g7" event={"ID":"0f588baf-81cd-4efb-a767-bdbd544e38d4","Type":"ContainerStarted","Data":"1cd54e3870779cb2874ebc874e464ad5a7a236f9badfcc714ebc312838c6b85d"}
Sep 17 15:30:29.606705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.606685    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vkkkv" event={"ID":"34369b40-593d-40a5-a244-cbcb4d0cea24","Type":"ContainerStarted","Data":"f39af3b5d3cd8386d5d0b1820f0897dcbc6cb78c1ef0b383440eef3c49da9969"}
Sep 17 15:30:29.608249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.608231    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft" event={"ID":"276aa7f1-1096-4e27-bb04-9e20d484d804","Type":"ContainerStarted","Data":"8555f0e1db0cf06b8e0fab70cc1a018f4b3fa2951ef78b764e426f4da1fd9aad"}
Sep 17 15:30:29.612708 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.612672    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cjwxc" podStartSLOduration=3.563768976 podStartE2EDuration="21.612661077s" totalImagesPullingTime="18.048892101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:29.612377587 +0000 UTC m=+21.579555406" watchObservedRunningTime="2026-09-17 15:30:29.612661077 +0000 UTC m=+21.579838895"
Sep 17 15:30:29.627842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.627804    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6kn45" podStartSLOduration=3.612896589 podStartE2EDuration="21.627794825s" totalImagesPullingTime="18.014898236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:29.627747756 +0000 UTC m=+21.594925589" watchObservedRunningTime="2026-09-17 15:30:29.627794825 +0000 UTC m=+21.594972643"
Sep 17 15:30:29.641818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.641778    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vbfj6" podStartSLOduration=7.660845356 podStartE2EDuration="21.64176663s" totalImagesPullingTime="13.980921274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:29.641492765 +0000 UTC m=+21.608670583" watchObservedRunningTime="2026-09-17 15:30:29.64176663 +0000 UTC m=+21.608944447"
Sep 17 15:30:29.643417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.643400    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cjwxc_7daf6886-a7af-4b98-834f-5dee8153323a/node-ca/0.log"
Sep 17 15:30:29.675602 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:29.675564    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6r2g7" podStartSLOduration=3.6556791520000003 podStartE2EDuration="21.675552987s" totalImagesPullingTime="18.019873835s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:29.675111027 +0000 UTC m=+21.642288848" watchObservedRunningTime="2026-09-17 15:30:29.675552987 +0000 UTC m=+21.642730804"
Sep 17 15:30:30.519337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.519316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:30.519469 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:30.519399    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:30.519469 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.519316    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:30.519591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.519318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:30.519591 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:30.519522    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:30.519591 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:30.519552    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:30.617056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.616972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"d64a75c23a1afaadceb8f6abf1a7004f3b9086c5c1f7e5aed03dd8cac695f257"}
Sep 17 15:30:30.617413 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.617057    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"f634e8c697288ccba3ef2690711c158a4955c1953fdae987a4215a32c4ba2196"}
Sep 17 15:30:30.618449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.618426    2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:30:30.618993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.618946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5pvpd" event={"ID":"9bc9c39f-56af-494c-b585-cb5f46588bc6","Type":"ContainerStarted","Data":"d49375433f876d0322eb4fbfd3f9c339d644749241156b61d24a22d1c36f39f7"}
Sep 17 15:30:30.634712 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.634666    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vkkkv" podStartSLOduration=4.544550966 podStartE2EDuration="22.63465066s" totalImagesPullingTime="18.090099694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:29.69212061 +0000 UTC m=+21.659298428" watchObservedRunningTime="2026-09-17 15:30:30.63465066 +0000 UTC m=+22.601828476"
Sep 17 15:30:30.634897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.634876    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5pvpd" podStartSLOduration=4.592408024 podStartE2EDuration="22.63487081s" totalImagesPullingTime="18.042462786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:30.634767763 +0000 UTC m=+22.601945581" watchObservedRunningTime="2026-09-17 15:30:30.63487081 +0000 UTC m=+22.602048628"
Sep 17 15:30:30.989046 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.989014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:30.989492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:30.989475    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:31.451865 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:31.451674    2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:30:30.618446391Z","UUID":"52b974b1-8f05-45ed-91be-0eac6571f616","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:30:31.453471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:31.453448    2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:30:31.453471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:31.453484    2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:30:31.623805 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:31.623766    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft" event={"ID":"276aa7f1-1096-4e27-bb04-9e20d484d804","Type":"ContainerStarted","Data":"5f3df4abd8b3aa7aff5e61ba8063ecd520d8317a99265612e0e310725284c7c0"}
Sep 17 15:30:32.519023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.518990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:32.519289 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.519104    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:32.519289 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:32.519110    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:32.519289 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:32.519230    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:32.519442 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.519292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:32.519442 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:32.519417    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:32.630127 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.630094    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"4cfcb1b0c0ace2c10d65467fda036a55f81476140325b449749ef3e1b91b4db0"}
Sep 17 15:30:32.632213 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.632182    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft" event={"ID":"276aa7f1-1096-4e27-bb04-9e20d484d804","Type":"ContainerStarted","Data":"61c120523348ddc3066222a7b858574efa0ec292df26a5c07221d1b2e8f75f2d"}
Sep 17 15:30:32.632213 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.632203    2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:30:32.654663 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:32.654604    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xrhft" podStartSLOduration=4.423575291 podStartE2EDuration="24.654592837s" totalImagesPullingTime="20.231017546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:32.654404463 +0000 UTC m=+24.621582286" watchObservedRunningTime="2026-09-17 15:30:32.654592837 +0000 UTC m=+24.621770657"
Sep 17 15:30:34.518605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:34.518573    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:34.518605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:34.518578    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:34.519091 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:34.518636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:34.519091 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:34.518735    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:34.519091 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:34.518812    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:34.519091 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:34.518865    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:35.149356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:35.149241    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:35.149822 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:35.149799    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vbfj6"
Sep 17 15:30:35.640569 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:35.640535    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerStarted","Data":"f22129f59656e82cc062889fec0dd53b9039e18e044cff5e5a3014e148a9e74a"}
Sep 17 15:30:35.644550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:35.644522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" event={"ID":"02bc3c35-2376-4376-a493-ad3f3b677faa","Type":"ContainerStarted","Data":"1d1d2e57a1ab6e9e1bf71b9245466dc935c254b4e73d1fdc753b6de4c8bfab58"}
Sep 17 15:30:35.696759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:35.696708    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7" podStartSLOduration=9.614954778 podStartE2EDuration="27.696689616s" totalImagesPullingTime="18.081734838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:35.696256722 +0000 UTC m=+27.663434539" watchObservedRunningTime="2026-09-17 15:30:35.696689616 +0000 UTC m=+27.663867435"
Sep 17 15:30:36.519309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.519277    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:36.519636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.519284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:36.519636 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:36.519371    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:36.519636 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:36.519453    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:36.519636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.519289    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:36.519636 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:36.519574    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:36.590395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.590373    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:36.647952 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.647928    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="f22129f59656e82cc062889fec0dd53b9039e18e044cff5e5a3014e148a9e74a" exitCode=0
Sep 17 15:30:36.648475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.648013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"f22129f59656e82cc062889fec0dd53b9039e18e044cff5e5a3014e148a9e74a"}
Sep 17 15:30:36.648536 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.648473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:36.648536 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.648523    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:36.663328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.663311    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:36.663414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:36.663391    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:30:38.519624 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:38.519585    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:38.519989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:38.519666    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:38.519989 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:38.519679    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:38.519989 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:38.519741    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:38.519989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:38.519781    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:38.519989 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:38.519890    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:38.653920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:38.653736    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="f24d9d4a08c4a081638ed42e26458c617371585bddd7969cf78713e7e21b9ad6" exitCode=0
Sep 17 15:30:38.654115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:38.653820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"f24d9d4a08c4a081638ed42e26458c617371585bddd7969cf78713e7e21b9ad6"}
Sep 17 15:30:40.519100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:40.519069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:40.519531 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:40.519069    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:40.519531 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:40.519179    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:40.519531 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:40.519070    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:40.519531 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:40.519287    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:40.519531 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:40.519334    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:40.660121 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:40.660094    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="6e178bf26949fdec45739681a3b6d33d7e4166f80f1e7bcf4574f0669d636644" exitCode=0
Sep 17 15:30:40.660277 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:40.660136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"6e178bf26949fdec45739681a3b6d33d7e4166f80f1e7bcf4574f0669d636644"}
Sep 17 15:30:42.139427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.139392    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:42.139427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.139431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:42.139944 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.139568    2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:42.139944 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.139641    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs podName:9ff08dc9-0de2-465c-b8c6-aca3b069fd04 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:14.139623143 +0000 UTC m=+66.106800951 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs") pod "network-metrics-daemon-8m8nk" (UID: "9ff08dc9-0de2-465c-b8c6-aca3b069fd04") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:30:42.139944 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.139568    2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:42.139944 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.139697    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret podName:c2cef1e7-01db-4378-ac73-9efa6ba71ee5 nodeName:}" failed. No retries permitted until 2026-09-17 15:31:14.139684981 +0000 UTC m=+66.106862782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret") pod "global-pull-secret-syncer-kg4qt" (UID: "c2cef1e7-01db-4378-ac73-9efa6ba71ee5") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:30:42.341338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.341305    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:42.341511 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.341459    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:30:42.341511 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.341477    2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:30:42.341511 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.341486    2564 projected.go:196] Error preparing data for projected volume kube-api-access-h7w4k for pod openshift-network-diagnostics/network-check-target-67sjn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:42.341701 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.341543    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k podName:96aac0ea-441d-4cfb-b9ca-31dc9855bbaf nodeName:}" failed. No retries permitted until 2026-09-17 15:31:14.341527986 +0000 UTC m=+66.308705794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-h7w4k" (UniqueName: "kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k") pod "network-check-target-67sjn" (UID: "96aac0ea-441d-4cfb-b9ca-31dc9855bbaf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:30:42.519228 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.519199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:42.519515 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.519199    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:42.519515 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.519335    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:42.519515 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:42.519355    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:42.519515 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.519435    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:42.519732 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:42.519514    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:44.518895 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:44.518856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:44.519377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:44.518856    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:44.519377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:44.518990    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:44.519377 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:44.518987    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:44.519377 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:44.519090    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:44.519377 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:44.519173    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:45.675943 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:45.675915    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerStarted","Data":"fbc42147dfaa90cd80fdf864037a21b1239fc47b1611157dae64efa11eda42fc"}
Sep 17 15:30:46.518703 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:46.518673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:46.518886 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:46.518768    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:46.518886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:46.518673    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:46.518886 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:46.518874    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:46.519052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:46.518918    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:46.519052 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:46.518968    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:46.681058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:46.681023    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="fbc42147dfaa90cd80fdf864037a21b1239fc47b1611157dae64efa11eda42fc" exitCode=0
Sep 17 15:30:46.681489 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:46.681071    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"fbc42147dfaa90cd80fdf864037a21b1239fc47b1611157dae64efa11eda42fc"}
Sep 17 15:30:47.688191 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:47.688160    2564 generic.go:356] "Generic (PLEG): container finished" podID="aab0eb57-6e0b-4891-8aa6-f20a9a5ec930" containerID="47b5330014b0ae593f7d470740eb3b4b3cac49831ad413768db54b1bc6a29bcf" exitCode=0
Sep 17 15:30:47.688542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:47.688209    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerDied","Data":"47b5330014b0ae593f7d470740eb3b4b3cac49831ad413768db54b1bc6a29bcf"}
Sep 17 15:30:48.519200 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:48.519169    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:48.519379 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:48.519252    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:48.519379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:48.519283    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:48.519379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:48.519284    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:48.519379 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:48.519354    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:48.519536 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:48.519447    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:48.693759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:48.693727    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c99tn" event={"ID":"aab0eb57-6e0b-4891-8aa6-f20a9a5ec930","Type":"ContainerStarted","Data":"9cd0618c1da55051888f47641e93b07081f28104b8aa2be56466819b2ed8508c"}
Sep 17 15:30:48.713308 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:48.713272    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c99tn" podStartSLOduration=9.698501037 podStartE2EDuration="40.713259084s" totalImagesPullingTime="31.014758047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:30:48.713162772 +0000 UTC m=+40.680340590" watchObservedRunningTime="2026-09-17 15:30:48.713259084 +0000 UTC m=+40.680436903"
Sep 17 15:30:50.519319 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:50.519286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:50.519319 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:50.519307    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:50.519764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:50.519286    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:50.519764 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:50.519385    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:50.519764 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:50.519455    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:50.519764 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:50.519510    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:52.519218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:52.519187    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:52.519874 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:52.519214    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:52.520042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:52.519309    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:52.520275 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:52.520253    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:52.520544 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:52.520489    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:52.520663 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:52.520633    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:53.220018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.219815    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67sjn"]
Sep 17 15:30:53.220162 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.220109    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:53.220238 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:53.220217    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:53.222547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.222514    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg4qt"]
Sep 17 15:30:53.222688 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.222600    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:53.222748 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:53.222696    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:53.223277 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.223252    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8m8nk"]
Sep 17 15:30:53.223379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:53.223343    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:53.223446 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:53.223427    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:54.518818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:54.518786    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:54.519206 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:54.518790    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:54.519206 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:54.518887    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:54.519206 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:54.518947    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:55.519324 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:55.519291    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:55.519755 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:55.519396    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:56.519015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:56.518980    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:56.519214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:56.518988    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:56.519214 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:56.519077    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kg4qt" podUID="c2cef1e7-01db-4378-ac73-9efa6ba71ee5"
Sep 17 15:30:56.519214 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:56.519145    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-67sjn" podUID="96aac0ea-441d-4cfb-b9ca-31dc9855bbaf"
Sep 17 15:30:57.519077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.519046    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:57.519602 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:30:57.519179    2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8m8nk" podUID="9ff08dc9-0de2-465c-b8c6-aca3b069fd04"
Sep 17 15:30:57.801823 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.801744    2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-17.ec2.internal" event="NodeReady"
Sep 17 15:30:57.801956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.801859    2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:30:57.846910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.846883    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xmxw6"]
Sep 17 15:30:57.848716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.848699    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.850352 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.850330    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6lz9k"]
Sep 17 15:30:57.851867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.851505    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:30:57.851867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.851517    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:30:57.851867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.851637    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l7hr4\""
Sep 17 15:30:57.852289 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.852270    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.854292 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.854273    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:30:57.854542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.854526    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hk45w\""
Sep 17 15:30:57.855137 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.855118    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:30:57.855237 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.855225    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:30:57.855304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.855257    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:30:57.858423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.858402    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmxw6"]
Sep 17 15:30:57.860209 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860186    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45wdm\" (UniqueName: \"kubernetes.io/projected/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-api-access-45wdm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.860302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/1319ecb9-3468-413d-b218-aedd3a513e45-kube-api-access-ltf6j\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.860360 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860343    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1319ecb9-3468-413d-b218-aedd3a513e45-tmp-dir\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.860414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860377    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32f974a-a7c4-43fa-bda6-dc430abacbad-crio-socket\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.860464 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860438    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1319ecb9-3468-413d-b218-aedd3a513e45-metrics-tls\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.860518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1319ecb9-3468-413d-b218-aedd3a513e45-config-volume\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.860570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860540    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32f974a-a7c4-43fa-bda6-dc430abacbad-data-volume\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.861117 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860585    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32f974a-a7c4-43fa-bda6-dc430abacbad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.861117 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.860643    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.864549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.864529    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6lz9k"]
Sep 17 15:30:57.867522 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.867501    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6bztp"]
Sep 17 15:30:57.869282 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.869266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:57.871515 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.871497    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bcwx7\""
Sep 17 15:30:57.871605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.871538    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:30:57.871744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.871727    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:30:57.871816 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.871757    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:30:57.880402 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.880382    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6bztp"]
Sep 17 15:30:57.961430 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961400    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45wdm\" (UniqueName: \"kubernetes.io/projected/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-api-access-45wdm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.961430 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/1319ecb9-3468-413d-b218-aedd3a513e45-kube-api-access-ltf6j\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.961640 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-cert\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:57.961640 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961568    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1319ecb9-3468-413d-b218-aedd3a513e45-tmp-dir\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.961640 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961585    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32f974a-a7c4-43fa-bda6-dc430abacbad-crio-socket\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.961640 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961625    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1319ecb9-3468-413d-b218-aedd3a513e45-metrics-tls\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.961827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1319ecb9-3468-413d-b218-aedd3a513e45-config-volume\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.961827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32f974a-a7c4-43fa-bda6-dc430abacbad-data-volume\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.961827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961687    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32f974a-a7c4-43fa-bda6-dc430abacbad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.961827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961711    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztdfk\" (UniqueName: \"kubernetes.io/projected/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-kube-api-access-ztdfk\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:57.961827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.962066 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961861    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b32f974a-a7c4-43fa-bda6-dc430abacbad-crio-socket\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.962066 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.961886    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1319ecb9-3468-413d-b218-aedd3a513e45-tmp-dir\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.962156 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.962078    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b32f974a-a7c4-43fa-bda6-dc430abacbad-data-volume\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.962349 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.962329    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1319ecb9-3468-413d-b218-aedd3a513e45-config-volume\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.962391 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.962354    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.965472 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.965452    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b32f974a-a7c4-43fa-bda6-dc430abacbad-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:57.965472 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.965467    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1319ecb9-3468-413d-b218-aedd3a513e45-metrics-tls\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.969163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.969139    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltf6j\" (UniqueName: \"kubernetes.io/projected/1319ecb9-3468-413d-b218-aedd3a513e45-kube-api-access-ltf6j\") pod \"dns-default-xmxw6\" (UID: \"1319ecb9-3468-413d-b218-aedd3a513e45\") " pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:57.969259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:57.969241    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45wdm\" (UniqueName: \"kubernetes.io/projected/b32f974a-a7c4-43fa-bda6-dc430abacbad-kube-api-access-45wdm\") pod \"insights-runtime-extractor-6lz9k\" (UID: \"b32f974a-a7c4-43fa-bda6-dc430abacbad\") " pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:58.062416 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.062356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-cert\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:58.062518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.062432    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztdfk\" (UniqueName: \"kubernetes.io/projected/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-kube-api-access-ztdfk\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:58.064479 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.064458    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-cert\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:58.069561 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.069543    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztdfk\" (UniqueName: \"kubernetes.io/projected/b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a-kube-api-access-ztdfk\") pod \"ingress-canary-6bztp\" (UID: \"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a\") " pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:58.164103 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.164084    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:30:58.169477 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.169459    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6lz9k"
Sep 17 15:30:58.176863 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.176844    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6bztp"
Sep 17 15:30:58.462321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.462236    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6lz9k"]
Sep 17 15:30:58.462864 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.462846    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xmxw6"]
Sep 17 15:30:58.463439 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.463339    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6bztp"]
Sep 17 15:30:58.466663 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:58.466630    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1319ecb9_3468_413d_b218_aedd3a513e45.slice/crio-e205c3628703338f90463085d295382800b72d973f342ab8887b7445f741e31a WatchSource:0}: Error finding container e205c3628703338f90463085d295382800b72d973f342ab8887b7445f741e31a: Status 404 returned error can't find the container with id e205c3628703338f90463085d295382800b72d973f342ab8887b7445f741e31a
Sep 17 15:30:58.467477 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:30:58.467394    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3bebf42_fc2f_4c92_b2ee_73b10cd40e0a.slice/crio-3c27b8bd5c99eaf2146c0bff5e4e25006ddb89e5e63b643247d85040fdc66758 WatchSource:0}: Error finding container 3c27b8bd5c99eaf2146c0bff5e4e25006ddb89e5e63b643247d85040fdc66758: Status 404 returned error can't find the container with id 3c27b8bd5c99eaf2146c0bff5e4e25006ddb89e5e63b643247d85040fdc66758
Sep 17 15:30:58.525730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.525708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:30:58.526283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.525734    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:30:58.527972 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.527955    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\""
Sep 17 15:30:58.528056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.527956    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:30:58.528056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.528000    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:30:58.528056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.528035    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:30:58.715915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.715836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6bztp" event={"ID":"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a","Type":"ContainerStarted","Data":"3c27b8bd5c99eaf2146c0bff5e4e25006ddb89e5e63b643247d85040fdc66758"}
Sep 17 15:30:58.717313 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.717291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lz9k" event={"ID":"b32f974a-a7c4-43fa-bda6-dc430abacbad","Type":"ContainerStarted","Data":"d6e8c64805609cd9d0ba34433cb4f4575c5c5e166d855dc948a166666dcf8d2c"}
Sep 17 15:30:58.717427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.717336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lz9k" event={"ID":"b32f974a-a7c4-43fa-bda6-dc430abacbad","Type":"ContainerStarted","Data":"2e427841826086cbd662c98beddde0738a4baae3c2066282cf69feb2e557efc4"}
Sep 17 15:30:58.718336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:58.718311    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmxw6" event={"ID":"1319ecb9-3468-413d-b218-aedd3a513e45","Type":"ContainerStarted","Data":"e205c3628703338f90463085d295382800b72d973f342ab8887b7445f741e31a"}
Sep 17 15:30:59.519033 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.519008    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:30:59.522070 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.521827    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\""
Sep 17 15:30:59.522070 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.521999    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:30:59.587082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.587052    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:30:59.589836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.589813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.592209 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592135    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4tjvv\""
Sep 17 15:30:59.592209 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592151    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:30:59.592403 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592327    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:30:59.592403 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592342    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:30:59.592499 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592454    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:30:59.592691 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592583    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:30:59.592859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592842    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:30:59.592969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.592891    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:30:59.600178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.600145    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:30:59.676395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676365    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.676567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676407    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.676567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.676705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.676705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676638    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g94n\" (UniqueName: \"kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.676806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.676701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.724127 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.724092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lz9k" event={"ID":"b32f974a-a7c4-43fa-bda6-dc430abacbad","Type":"ContainerStarted","Data":"751799f3f79f45a47ca8312dd87de7de9a3ae6df5be5903dc115fd95973fb2e1"}
Sep 17 15:30:59.777509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777444    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.777509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.777730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777521    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g94n\" (UniqueName: \"kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.777730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777665    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.777730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777723    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.777891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.777748    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.778331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.778305    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.778716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.778694    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.779052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.779007    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.781367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.781344    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.781448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.781362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.786251 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.786231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g94n\" (UniqueName: \"kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n\") pod \"console-7bcb6795d6-kxl7s\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") " pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:30:59.902136 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:30:59.902098    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:01.502369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.501964    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:31:01.507172 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:01.507144    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f2bae8_c94f_4b3d_b2b6_e73f383766d0.slice/crio-28375c35f0aec4ed9baeeea1ad34b3d43ce5f74d2fde4c69823ca6e2825dd844 WatchSource:0}: Error finding container 28375c35f0aec4ed9baeeea1ad34b3d43ce5f74d2fde4c69823ca6e2825dd844: Status 404 returned error can't find the container with id 28375c35f0aec4ed9baeeea1ad34b3d43ce5f74d2fde4c69823ca6e2825dd844
Sep 17 15:31:01.731945 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.731904    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6lz9k" event={"ID":"b32f974a-a7c4-43fa-bda6-dc430abacbad","Type":"ContainerStarted","Data":"762ae787c11b82b3c7169e0377778c5ae9f2608ceef9c5536f1672494d5cee97"}
Sep 17 15:31:01.733214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.733178    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb6795d6-kxl7s" event={"ID":"21f2bae8-c94f-4b3d-b2b6-e73f383766d0","Type":"ContainerStarted","Data":"28375c35f0aec4ed9baeeea1ad34b3d43ce5f74d2fde4c69823ca6e2825dd844"}
Sep 17 15:31:01.734879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.734853    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmxw6" event={"ID":"1319ecb9-3468-413d-b218-aedd3a513e45","Type":"ContainerStarted","Data":"bea63f0b8a108058e65c34adfc69b637193897cc78d579a2ffc59aee417c0158"}
Sep 17 15:31:01.735001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.734901    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xmxw6" event={"ID":"1319ecb9-3468-413d-b218-aedd3a513e45","Type":"ContainerStarted","Data":"982ff45545d59ad3722c69a3d7f8978c69cb25248a638dc23fe2dcc1013cfc78"}
Sep 17 15:31:01.735001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.734929    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:31:01.736259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.736232    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6bztp" event={"ID":"b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a","Type":"ContainerStarted","Data":"f021da033174c801982ea0276c6fde6d825f9aa7a01afc0396721f8214a5cb48"}
Sep 17 15:31:01.754104 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.753981    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6lz9k" podStartSLOduration=2.095461174 podStartE2EDuration="4.75396539s" totalImagesPullingTime="2.658504216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.753913543 +0000 UTC m=+53.721091360" watchObservedRunningTime="2026-09-17 15:31:01.75396539 +0000 UTC m=+53.721143198"
Sep 17 15:31:01.772029 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.771972    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6bztp" podStartSLOduration=2.003808331 podStartE2EDuration="4.771953784s" totalImagesPullingTime="2.768145453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.771263964 +0000 UTC m=+53.738441797" watchObservedRunningTime="2026-09-17 15:31:01.771953784 +0000 UTC m=+53.739131601"
Sep 17 15:31:01.801036 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:01.800801    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xmxw6" podStartSLOduration=2.042192351 podStartE2EDuration="4.800780645s" totalImagesPullingTime="2.758588294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:01.80003534 +0000 UTC m=+53.767213159" watchObservedRunningTime="2026-09-17 15:31:01.800780645 +0000 UTC m=+53.767958466"
Sep 17 15:31:04.752668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:04.752569    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb6795d6-kxl7s" event={"ID":"21f2bae8-c94f-4b3d-b2b6-e73f383766d0","Type":"ContainerStarted","Data":"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"}
Sep 17 15:31:04.770327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:04.770284    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bcb6795d6-kxl7s" podStartSLOduration=2.885723052 podStartE2EDuration="5.770272533s" totalImagesPullingTime="2.884549481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:04.769515282 +0000 UTC m=+56.736693100" watchObservedRunningTime="2026-09-17 15:31:04.770272533 +0000 UTC m=+56.737450345"
Sep 17 15:31:05.333731 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.333696    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"]
Sep 17 15:31:05.335662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.335646    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.338420 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338352    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:31:05.338578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338428    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:31:05.338578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338442    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:31:05.338578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338445    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jltzx\""
Sep 17 15:31:05.338578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338531    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:31:05.338806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.338590    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:31:05.347390 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.347367    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"]
Sep 17 15:31:05.421897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.421871    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.422020 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.421902    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.422020 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.421926    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktlkq\" (UniqueName: \"kubernetes.io/projected/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-kube-api-access-ktlkq\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.422020 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.421982    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.522535 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.522506    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.522668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.522542    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktlkq\" (UniqueName: \"kubernetes.io/projected/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-kube-api-access-ktlkq\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.522668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.522562    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.522668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.522653    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.523494 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.523475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.525882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.525864    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.525972 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.525954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.531188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.531166    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktlkq\" (UniqueName: \"kubernetes.io/projected/9f6537c4-f5a0-47c0-81d6-b13eebc305ab-kube-api-access-ktlkq\") pod \"prometheus-operator-68bcd7f987-6hft9\" (UID: \"9f6537c4-f5a0-47c0-81d6-b13eebc305ab\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.644934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.644872    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"
Sep 17 15:31:05.913683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:05.913595    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-6hft9"]
Sep 17 15:31:05.917327 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:05.917301    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f6537c4_f5a0_47c0_81d6_b13eebc305ab.slice/crio-5a298d8c2fd8d9bf7611e3a37346ce8bfdb9a27e1c8fe2d9eb6c2c5da1cfc826 WatchSource:0}: Error finding container 5a298d8c2fd8d9bf7611e3a37346ce8bfdb9a27e1c8fe2d9eb6c2c5da1cfc826: Status 404 returned error can't find the container with id 5a298d8c2fd8d9bf7611e3a37346ce8bfdb9a27e1c8fe2d9eb6c2c5da1cfc826
Sep 17 15:31:06.762301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:06.762266    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9" event={"ID":"9f6537c4-f5a0-47c0-81d6-b13eebc305ab","Type":"ContainerStarted","Data":"5a298d8c2fd8d9bf7611e3a37346ce8bfdb9a27e1c8fe2d9eb6c2c5da1cfc826"}
Sep 17 15:31:07.767591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:07.767549    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9" event={"ID":"9f6537c4-f5a0-47c0-81d6-b13eebc305ab","Type":"ContainerStarted","Data":"7071137294af56c45d14ee7b089d9d7b59700331ab5d3b5f18df032b37be4129"}
Sep 17 15:31:07.767991 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:07.767631    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9" event={"ID":"9f6537c4-f5a0-47c0-81d6-b13eebc305ab","Type":"ContainerStarted","Data":"83f07e9fd0d58724745bbfa0c32db4ffcd1d38220ed38c385476ca448807e05f"}
Sep 17 15:31:07.783900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:07.783859    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-6hft9" podStartSLOduration=1.595638284 podStartE2EDuration="2.783844413s" totalImagesPullingTime="1.188206129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:07.783578302 +0000 UTC m=+59.750756119" watchObservedRunningTime="2026-09-17 15:31:07.783844413 +0000 UTC m=+59.751022231"
Sep 17 15:31:08.673878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:08.673851    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jp8g7"
Sep 17 15:31:09.728235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.728202    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p9f9p"]
Sep 17 15:31:09.730846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.730827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.733603 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.733576    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jw4fs\""
Sep 17 15:31:09.733913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.733896    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:31:09.734893 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.734873    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:31:09.734969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.734905    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:31:09.754897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.754872    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-textfile\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.754993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.754900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-metrics-client-ca\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.754993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.754920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-tls\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-root\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-sys\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-wtmp\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.755241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.755172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v78z\" (UniqueName: \"kubernetes.io/projected/612681ab-52a7-44bc-8b80-c60399813b50-kube-api-access-2v78z\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856376 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856349    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-textfile\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856383    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-metrics-client-ca\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856411    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856459    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-tls\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-root\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-sys\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856558    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-wtmp\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856632    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v78z\" (UniqueName: \"kubernetes.io/projected/612681ab-52a7-44bc-8b80-c60399813b50-kube-api-access-2v78z\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856659    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-root\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-sys\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856730    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-textfile\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.856969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.856802    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-wtmp\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.857182 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.857158    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-accelerators-collector-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.857224 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.857163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/612681ab-52a7-44bc-8b80-c60399813b50-metrics-client-ca\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.858767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.858751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-tls\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.858845 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.858787    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/612681ab-52a7-44bc-8b80-c60399813b50-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.868682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.868654    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v78z\" (UniqueName: \"kubernetes.io/projected/612681ab-52a7-44bc-8b80-c60399813b50-kube-api-access-2v78z\") pod \"node-exporter-p9f9p\" (UID: \"612681ab-52a7-44bc-8b80-c60399813b50\") " pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:09.902643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.902602    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:09.902711 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.902706    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:09.908052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:09.908032    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:10.039079 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.039050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p9f9p"
Sep 17 15:31:10.048067 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:10.048046    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612681ab_52a7_44bc_8b80_c60399813b50.slice/crio-02acc44c81dc0dccc12e351163e321e32a812e7c8d80d911331904a7103eb6b7 WatchSource:0}: Error finding container 02acc44c81dc0dccc12e351163e321e32a812e7c8d80d911331904a7103eb6b7: Status 404 returned error can't find the container with id 02acc44c81dc0dccc12e351163e321e32a812e7c8d80d911331904a7103eb6b7
Sep 17 15:31:10.784230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.784193    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9f9p" event={"ID":"612681ab-52a7-44bc-8b80-c60399813b50","Type":"ContainerStarted","Data":"02acc44c81dc0dccc12e351163e321e32a812e7c8d80d911331904a7103eb6b7"}
Sep 17 15:31:10.789893 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.789867    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:10.861323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.861167    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:31:10.864709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.864679    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.867201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.867174    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:31:10.867373 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.867174    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:31:10.868158 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.868087    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z5mx8\""
Sep 17 15:31:10.873537 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.873504    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:31:10.873704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.873682    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:31:10.873798 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.873713    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:31:10.873798 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.873760    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:31:10.874328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.874262    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:31:10.874414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.874370    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:31:10.878737 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.878719    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:31:10.916276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.916233    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:31:10.964447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964409    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-web-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964449    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964536    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964573    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964677    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964744    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6r6\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-kube-api-access-tc6r6\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964803    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.964920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:10.965119 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:10.964921    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-out\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065452    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065558 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065482    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-out\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065558 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065527    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-web-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065558 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065551    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065606    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065651    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065728    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065757    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065792    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6r6\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-kube-api-access-tc6r6\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.065851    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.065982 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:11.065936    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle podName:b50f3821-df72-4c91-b5f0-0420ee98a03c nodeName:}" failed. No retries permitted until 2026-09-17 15:31:11.565911441 +0000 UTC m=+63.533089263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "b50f3821-df72-4c91-b5f0-0420ee98a03c") : configmap references non-existent config key: ca-bundle.crt
Sep 17 15:31:11.066237 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.066200    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.068428 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:11.066305    2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 15:31:11.068428 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:11.066367    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls podName:b50f3821-df72-4c91-b5f0-0420ee98a03c nodeName:}" failed. No retries permitted until 2026-09-17 15:31:11.566349381 +0000 UTC m=+63.533527183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b50f3821-df72-4c91-b5f0-0420ee98a03c") : secret "alertmanager-main-tls" not found
Sep 17 15:31:11.069826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069018    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.069826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-volume\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.069826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b50f3821-df72-4c91-b5f0-0420ee98a03c-config-out\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.069826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069677    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.069826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069751    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.070178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.069995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.070682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.070643    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.071846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.071827    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-web-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.072050 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.072032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.080663 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.080646    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6r6\" (UniqueName: \"kubernetes.io/projected/b50f3821-df72-4c91-b5f0-0420ee98a03c-kube-api-access-tc6r6\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.569208 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.569178    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.569355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.569227    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.569355 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:11.569323    2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 15:31:11.569418 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:11.569375    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls podName:b50f3821-df72-4c91-b5f0-0420ee98a03c nodeName:}" failed. No retries permitted until 2026-09-17 15:31:12.569359712 +0000 UTC m=+64.536537508 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "b50f3821-df72-4c91-b5f0-0420ee98a03c") : secret "alertmanager-main-tls" not found
Sep 17 15:31:11.570142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.570119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b50f3821-df72-4c91-b5f0-0420ee98a03c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:11.747163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.747138    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xmxw6"
Sep 17 15:31:11.789112 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.789079    2564 generic.go:356] "Generic (PLEG): container finished" podID="612681ab-52a7-44bc-8b80-c60399813b50" containerID="bf352b6c2a3026530245569aa42b8fe36d21f656081943aee7b0ed3ae5d8c3dc" exitCode=0
Sep 17 15:31:11.789493 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:11.789167    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9f9p" event={"ID":"612681ab-52a7-44bc-8b80-c60399813b50","Type":"ContainerDied","Data":"bf352b6c2a3026530245569aa42b8fe36d21f656081943aee7b0ed3ae5d8c3dc"}
Sep 17 15:31:12.577882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.577850    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:12.580226 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.580202    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b50f3821-df72-4c91-b5f0-0420ee98a03c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b50f3821-df72-4c91-b5f0-0420ee98a03c\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:12.676130 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.676096    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:31:12.713525 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.713492    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7567c9f887-rqvrf"]
Sep 17 15:31:12.716339 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.716318    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.718969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.718940    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-d7grd5u3d0pdq\""
Sep 17 15:31:12.719079 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.719004    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2kvxn\""
Sep 17 15:31:12.719492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.719440    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:31:12.719940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.719591    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:31:12.719940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.719677    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:31:12.719940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.719704    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:31:12.720329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.720076    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:31:12.734675 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.734649    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7567c9f887-rqvrf"]
Sep 17 15:31:12.778787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.778923 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778792    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb56917-1441-414c-8e07-87181ad6835e-metrics-client-ca\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.778923 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778817    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-grpc-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.778923 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.779034 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778949    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.779034 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778976    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.779034 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.778997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.779171 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.779080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv26m\" (UniqueName: \"kubernetes.io/projected/fdb56917-1441-414c-8e07-87181ad6835e-kube-api-access-hv26m\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.794202 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.794177    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9f9p" event={"ID":"612681ab-52a7-44bc-8b80-c60399813b50","Type":"ContainerStarted","Data":"11075a9446daaa42792d2ec1bb0e7489a36ee42f591f3ed0d7ef10c7f3dcb28a"}
Sep 17 15:31:12.794540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.794228    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p9f9p" event={"ID":"612681ab-52a7-44bc-8b80-c60399813b50","Type":"ContainerStarted","Data":"dedcdf5a2fc6898a1b59d4a2d9a128094fe83dc9178dc7380d3587d2ab500939"}
Sep 17 15:31:12.812655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.812597    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p9f9p" podStartSLOduration=2.75275931 podStartE2EDuration="3.812584821s" totalImagesPullingTime="999.580062ms" totalInitContainerRuntime="60.245449ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:12.812194803 +0000 UTC m=+64.779372620" watchObservedRunningTime="2026-09-17 15:31:12.812584821 +0000 UTC m=+64.779762639"
Sep 17 15:31:12.879928 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.879864    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.879928 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.879900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb56917-1441-414c-8e07-87181ad6835e-metrics-client-ca\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.879933    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-grpc-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.879996    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.880075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.880100    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.880128    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.880187    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv26m\" (UniqueName: \"kubernetes.io/projected/fdb56917-1441-414c-8e07-87181ad6835e-kube-api-access-hv26m\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.880705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.880680    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fdb56917-1441-414c-8e07-87181ad6835e-metrics-client-ca\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.882793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.882772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.882967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.882940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.883036 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.883015    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.883036 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.883028    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-grpc-tls\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.883390 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.883370    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.883468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.883407    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fdb56917-1441-414c-8e07-87181ad6835e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.888243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.888228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv26m\" (UniqueName: \"kubernetes.io/projected/fdb56917-1441-414c-8e07-87181ad6835e-kube-api-access-hv26m\") pod \"thanos-querier-7567c9f887-rqvrf\" (UID: \"fdb56917-1441-414c-8e07-87181ad6835e\") " pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:12.959575 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:12.959542    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:31:12.962357 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:12.962326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50f3821_df72_4c91_b5f0_0420ee98a03c.slice/crio-e9f845e793782d230c24c701e2a592bb2037fa51f0277c7f285ab1eb799808eb WatchSource:0}: Error finding container e9f845e793782d230c24c701e2a592bb2037fa51f0277c7f285ab1eb799808eb: Status 404 returned error can't find the container with id e9f845e793782d230c24c701e2a592bb2037fa51f0277c7f285ab1eb799808eb
Sep 17 15:31:13.033507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:13.033475    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:13.324062 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:13.324028    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7567c9f887-rqvrf"]
Sep 17 15:31:13.327664 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:13.327634    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb56917_1441_414c_8e07_87181ad6835e.slice/crio-3bfdda3edd352334cc59f09f26734a1ae810e182d3d4eb320c46875189213d07 WatchSource:0}: Error finding container 3bfdda3edd352334cc59f09f26734a1ae810e182d3d4eb320c46875189213d07: Status 404 returned error can't find the container with id 3bfdda3edd352334cc59f09f26734a1ae810e182d3d4eb320c46875189213d07
Sep 17 15:31:13.799235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:13.799199    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"3bfdda3edd352334cc59f09f26734a1ae810e182d3d4eb320c46875189213d07"}
Sep 17 15:31:13.800673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:13.800646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"e9f845e793782d230c24c701e2a592bb2037fa51f0277c7f285ab1eb799808eb"}
Sep 17 15:31:14.067470 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.067381    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57c57d476b-mccw5"]
Sep 17 15:31:14.069778 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.069759    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.072147 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.072124    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 15:31:14.073374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.073104    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:31:14.073374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.073157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e1hkopdahek7e\""
Sep 17 15:31:14.073374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.073156    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 15:31:14.073374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.073221    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-j9k4b\""
Sep 17 15:31:14.073374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.073267    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 15:31:14.080336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.080318    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c57d476b-mccw5"]
Sep 17 15:31:14.192540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192517    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxp9f\" (UniqueName: \"kubernetes.io/projected/6bb30e77-f930-4311-97a9-602ff4cdb5a4-kube-api-access-fxp9f\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192678 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192559    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-tls\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192678 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192634    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-client-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192778 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:31:14.192778 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192705    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:31:14.192778 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192742    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6bb30e77-f930-4311-97a9-602ff4cdb5a4-audit-log\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-metrics-server-audit-profiles\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192808    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.192913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.192849    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-client-certs\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.195306 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.195286    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:31:14.195409 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.195286    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:31:14.205304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.205276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c2cef1e7-01db-4378-ac73-9efa6ba71ee5-original-pull-secret\") pod \"global-pull-secret-syncer-kg4qt\" (UID: \"c2cef1e7-01db-4378-ac73-9efa6ba71ee5\") " pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:31:14.205465 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.205449    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9ff08dc9-0de2-465c-b8c6-aca3b069fd04-metrics-certs\") pod \"network-metrics-daemon-8m8nk\" (UID: \"9ff08dc9-0de2-465c-b8c6-aca3b069fd04\") " pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:31:14.232449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.232429    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pmg9m\""
Sep 17 15:31:14.241257 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.241233    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8m8nk"
Sep 17 15:31:14.296218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296165    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-tls\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296225    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-client-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296298    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6bb30e77-f930-4311-97a9-602ff4cdb5a4-audit-log\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296333    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-metrics-server-audit-profiles\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296399    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-client-certs\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.296591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.296454    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp9f\" (UniqueName: \"kubernetes.io/projected/6bb30e77-f930-4311-97a9-602ff4cdb5a4-kube-api-access-fxp9f\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.297630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.297569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/6bb30e77-f930-4311-97a9-602ff4cdb5a4-audit-log\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.299188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.299135    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.299800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.299760    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-tls\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.300868 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.300439    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/6bb30e77-f930-4311-97a9-602ff4cdb5a4-metrics-server-audit-profiles\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.302340 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.302299    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-client-ca-bundle\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.303168 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.303147    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/6bb30e77-f930-4311-97a9-602ff4cdb5a4-secret-metrics-server-client-certs\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.305914 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.305861    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxp9f\" (UniqueName: \"kubernetes.io/projected/6bb30e77-f930-4311-97a9-602ff4cdb5a4-kube-api-access-fxp9f\") pod \"metrics-server-57c57d476b-mccw5\" (UID: \"6bb30e77-f930-4311-97a9-602ff4cdb5a4\") " pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.382977 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.382914    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:14.396985 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.396960    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:31:14.399414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.399390    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:31:14.409583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.409562    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:31:14.419551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.419530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7w4k\" (UniqueName: \"kubernetes.io/projected/96aac0ea-441d-4cfb-b9ca-31dc9855bbaf-kube-api-access-h7w4k\") pod \"network-check-target-67sjn\" (UID: \"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf\") " pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:31:14.434940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.434915    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kg4qt"
Sep 17 15:31:14.441415 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.441339    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zrbtw\""
Sep 17 15:31:14.449522 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.449497    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:31:14.482367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.482329    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"]
Sep 17 15:31:14.504281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.504258    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"]
Sep 17 15:31:14.504412 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.504371    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.506350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.506323    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:31:14.506709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.506692    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cwzz5\""
Sep 17 15:31:14.529315 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.529254    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8m8nk"]
Sep 17 15:31:14.530790 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:14.530730    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff08dc9_0de2_465c_b8c6_aca3b069fd04.slice/crio-c3bfb8197e6e46538774d64600a8e903cb9d0cea89d6e2bcd12f612be91f178f WatchSource:0}: Error finding container c3bfb8197e6e46538774d64600a8e903cb9d0cea89d6e2bcd12f612be91f178f: Status 404 returned error can't find the container with id c3bfb8197e6e46538774d64600a8e903cb9d0cea89d6e2bcd12f612be91f178f
Sep 17 15:31:14.598128 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.598094    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.598258 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.598140    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58jrz\" (UniqueName: \"kubernetes.io/projected/7ef6441c-05be-4893-9f4b-97e7a8936a3e-kube-api-access-58jrz\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.661137 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.661093    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57c57d476b-mccw5"]
Sep 17 15:31:14.663471 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:14.663435    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb30e77_f930_4311_97a9_602ff4cdb5a4.slice/crio-7d78c2d0396d01762cfe68c7de13ca668303ffd66951d82e7985fba1d8dcfb8a WatchSource:0}: Error finding container 7d78c2d0396d01762cfe68c7de13ca668303ffd66951d82e7985fba1d8dcfb8a: Status 404 returned error can't find the container with id 7d78c2d0396d01762cfe68c7de13ca668303ffd66951d82e7985fba1d8dcfb8a
Sep 17 15:31:14.699312 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.699291    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58jrz\" (UniqueName: \"kubernetes.io/projected/7ef6441c-05be-4893-9f4b-97e7a8936a3e-kube-api-access-58jrz\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.699465 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.699362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.699528 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:14.699466    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:31:14.699528 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:14.699519    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert podName:7ef6441c-05be-4893-9f4b-97e7a8936a3e nodeName:}" failed. No retries permitted until 2026-09-17 15:31:15.199501835 +0000 UTC m=+67.166679633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-5v27j" (UID: "7ef6441c-05be-4893-9f4b-97e7a8936a3e") : secret "monitoring-plugin-cert" not found
Sep 17 15:31:14.709498 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.709472    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58jrz\" (UniqueName: \"kubernetes.io/projected/7ef6441c-05be-4893-9f4b-97e7a8936a3e-kube-api-access-58jrz\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:14.745913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.745848    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kg4qt"]
Sep 17 15:31:14.767362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.767338    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-67sjn"]
Sep 17 15:31:14.809979 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.809936    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5" event={"ID":"6bb30e77-f930-4311-97a9-602ff4cdb5a4","Type":"ContainerStarted","Data":"7d78c2d0396d01762cfe68c7de13ca668303ffd66951d82e7985fba1d8dcfb8a"}
Sep 17 15:31:14.811767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.811742    2564 generic.go:356] "Generic (PLEG): container finished" podID="b50f3821-df72-4c91-b5f0-0420ee98a03c" containerID="29f5c05ae20abf35df0338bee547060c3a0b39a29cb27ff24b947fce36cc1337" exitCode=0
Sep 17 15:31:14.811854 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.811834    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerDied","Data":"29f5c05ae20abf35df0338bee547060c3a0b39a29cb27ff24b947fce36cc1337"}
Sep 17 15:31:14.813184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:14.813153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8m8nk" event={"ID":"9ff08dc9-0de2-465c-b8c6-aca3b069fd04","Type":"ContainerStarted","Data":"c3bfb8197e6e46538774d64600a8e903cb9d0cea89d6e2bcd12f612be91f178f"}
Sep 17 15:31:15.202946 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:15.202911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:15.203146 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:15.203079    2564 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:31:15.203208 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:15.203162    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert podName:7ef6441c-05be-4893-9f4b-97e7a8936a3e nodeName:}" failed. No retries permitted until 2026-09-17 15:31:16.203140931 +0000 UTC m=+68.170318747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-5v27j" (UID: "7ef6441c-05be-4893-9f4b-97e7a8936a3e") : secret "monitoring-plugin-cert" not found
Sep 17 15:31:15.292206 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:15.292176    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96aac0ea_441d_4cfb_b9ca_31dc9855bbaf.slice/crio-ff8e7f4a35399d58c3c6870d5daae5d3cb103e060dde466265e4edf415e433c3 WatchSource:0}: Error finding container ff8e7f4a35399d58c3c6870d5daae5d3cb103e060dde466265e4edf415e433c3: Status 404 returned error can't find the container with id ff8e7f4a35399d58c3c6870d5daae5d3cb103e060dde466265e4edf415e433c3
Sep 17 15:31:15.293155 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:15.293049    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cef1e7_01db_4378_ac73_9efa6ba71ee5.slice/crio-8a2ec43e91eb54a7a5d745d0cd8f8b017a0182ba621fa41042ffff220ccfdd0c WatchSource:0}: Error finding container 8a2ec43e91eb54a7a5d745d0cd8f8b017a0182ba621fa41042ffff220ccfdd0c: Status 404 returned error can't find the container with id 8a2ec43e91eb54a7a5d745d0cd8f8b017a0182ba621fa41042ffff220ccfdd0c
Sep 17 15:31:15.821073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:15.821036    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"12396bda7a14a65ed99a23afefe80da43912a9e7c96996df6bd94046d51e2ef3"}
Sep 17 15:31:15.824431 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:15.824403    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67sjn" event={"ID":"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf","Type":"ContainerStarted","Data":"ff8e7f4a35399d58c3c6870d5daae5d3cb103e060dde466265e4edf415e433c3"}
Sep 17 15:31:15.826750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:15.826678    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg4qt" event={"ID":"c2cef1e7-01db-4378-ac73-9efa6ba71ee5","Type":"ContainerStarted","Data":"8a2ec43e91eb54a7a5d745d0cd8f8b017a0182ba621fa41042ffff220ccfdd0c"}
Sep 17 15:31:16.013637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.012760    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:31:16.029759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.029369    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.033947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.033861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:31:16.034218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.034200    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.035832    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.034865    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.034952    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.036259    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.034543    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.035050    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.036894    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.035005    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.037233    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.037297    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.037438    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4dfjk\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.037746    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:31:16.038157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.037890    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5nblmiann7pn8\""
Sep 17 15:31:16.040454 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.040123    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:31:16.042154 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.042077    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109254    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkh9p\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-kube-api-access-bkh9p\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109284    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109317    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109342    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109381    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109406    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109447    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109496    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-web-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109587    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109648    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config-out\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109674    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109707    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109737    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.110955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.109791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210537    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210576    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-web-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210609    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210657    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210684    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config-out\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.210800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210801    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210835    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210863    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210891    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210927    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210964    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.210987    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkh9p\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-kube-api-access-bkh9p\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.211020    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.211049    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.211074    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.211121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.211254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.211177    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.214917 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.214715    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.214917 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.214867    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.217319 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.215483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.217319 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.216902    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.217500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.217331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.218950 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.218626    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.218950 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.218700    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.220521 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.220497    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.227910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.227674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.227910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.227813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.228113    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config-out\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.228223    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.228508    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.228762    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.228987    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7ef6441c-05be-4893-9f4b-97e7a8936a3e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5v27j\" (UID: \"7ef6441c-05be-4893-9f4b-97e7a8936a3e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.229510    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.229921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-web-config\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.229982    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.229992    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.230400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.230331    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.234152 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.234071    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkh9p\" (UniqueName: \"kubernetes.io/projected/218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5-kube-api-access-bkh9p\") pod \"prometheus-k8s-0\" (UID: \"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.317633 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.317541    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:16.347585 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.347555    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:16.836259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.836121    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"823bd463a6d76858fd207a0864bf4143ec171e4a307ec033eb2bd5cbec106e3f"}
Sep 17 15:31:16.836259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:16.836207    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"17c789821cb0e73e630d050b2aae6473ad72cb994745733ada726c57337e3744"}
Sep 17 15:31:18.170443 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.170413    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"]
Sep 17 15:31:18.178605 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:18.177132    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef6441c_05be_4893_9f4b_97e7a8936a3e.slice/crio-5f97079a9172002fb569531900feb4ec2ab7399c40a8a02bcc3f2e3c21234cad WatchSource:0}: Error finding container 5f97079a9172002fb569531900feb4ec2ab7399c40a8a02bcc3f2e3c21234cad: Status 404 returned error can't find the container with id 5f97079a9172002fb569531900feb4ec2ab7399c40a8a02bcc3f2e3c21234cad
Sep 17 15:31:18.184104 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:31:18.184077    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218ffadf_a8ba_4f8c_bb96_5a9bb0d33da5.slice/crio-0d6d8cfe03514eafb10773f3fb3ff1f3a0191859cba07b98109bd93dbba43cec WatchSource:0}: Error finding container 0d6d8cfe03514eafb10773f3fb3ff1f3a0191859cba07b98109bd93dbba43cec: Status 404 returned error can't find the container with id 0d6d8cfe03514eafb10773f3fb3ff1f3a0191859cba07b98109bd93dbba43cec
Sep 17 15:31:18.184883 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.184830    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:31:18.846800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.846762    2564 generic.go:356] "Generic (PLEG): container finished" podID="218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5" containerID="63224817afad3dd340e8fe10e03f7e8c63301c4b85b989a9dce11cff133fa943" exitCode=0
Sep 17 15:31:18.846998 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.846850    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerDied","Data":"63224817afad3dd340e8fe10e03f7e8c63301c4b85b989a9dce11cff133fa943"}
Sep 17 15:31:18.846998 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.846962    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"0d6d8cfe03514eafb10773f3fb3ff1f3a0191859cba07b98109bd93dbba43cec"}
Sep 17 15:31:18.849219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.849189    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5" event={"ID":"6bb30e77-f930-4311-97a9-602ff4cdb5a4","Type":"ContainerStarted","Data":"1d5a4900146ae6a0666cdb4a615398360215f44631515aef5f187ddcb09e5aa9"}
Sep 17 15:31:18.853222 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.853185    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"2e96d4d78c8aef5b6ca67d2a7c296acaacd59074a95ad4796418d79ce4453ce5"}
Sep 17 15:31:18.853343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.853249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"daf1f8de0ea7e8575d2481a9ad7caddb2ee3c519b93deb9bc8e9be47d24eadc5"}
Sep 17 15:31:18.853343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.853294    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:18.853343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.853327    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" event={"ID":"fdb56917-1441-414c-8e07-87181ad6835e","Type":"ContainerStarted","Data":"c30e6a07ef7113b999ca02d4eda67795e211afca4ed8b93e7b7d9bd78a27fb0c"}
Sep 17 15:31:18.858138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858117    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"7b66f79e83fa401b5b089663825be90f280237df07d671f19df863fe230792a2"}
Sep 17 15:31:18.858355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858336    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"ae206cd76e6fa897849e403a186040d01ecdac8324fae85f29b556f35aa45400"}
Sep 17 15:31:18.858530 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858505    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"1d4b6159ed68738443e15587c8858c339f862b4defafe7dbf5647be3cd45b98d"}
Sep 17 15:31:18.858667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858558    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"553007891d4e427d964f5a3206c978f3ccb8b50fe43ff3db765123987d864336"}
Sep 17 15:31:18.858667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858597    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"b0a05bf56f4677f28930302afbc4a1db330b2630eb8ab308dafc8d9d94bf2b1b"}
Sep 17 15:31:18.858667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.858658    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b50f3821-df72-4c91-b5f0-0420ee98a03c","Type":"ContainerStarted","Data":"47c032457e5a87354ac3a7c66a675f588026e22846e1016c738c4fb0593df174"}
Sep 17 15:31:18.860592 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.860568    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8m8nk" event={"ID":"9ff08dc9-0de2-465c-b8c6-aca3b069fd04","Type":"ContainerStarted","Data":"2411b792b74d4e207e3c3385ee582dcf9c7c19f793919b7c599a6252f1f93a1c"}
Sep 17 15:31:18.860706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.860637    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8m8nk" event={"ID":"9ff08dc9-0de2-465c-b8c6-aca3b069fd04","Type":"ContainerStarted","Data":"cda2efeb10c6fa558687650a1dd101dc1436c23f1db87ddcf015ffd8f14a93f1"}
Sep 17 15:31:18.861908 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.861887    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j" event={"ID":"7ef6441c-05be-4893-9f4b-97e7a8936a3e","Type":"ContainerStarted","Data":"5f97079a9172002fb569531900feb4ec2ab7399c40a8a02bcc3f2e3c21234cad"}
Sep 17 15:31:18.904302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.904239    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8m8nk" podStartSLOduration=67.56923697 podStartE2EDuration="1m10.90421839s" totalImagesPullingTime="3.33498142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:18.903713315 +0000 UTC m=+70.870891133" watchObservedRunningTime="2026-09-17 15:31:18.90421839 +0000 UTC m=+70.871396207"
Sep 17 15:31:18.937459 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.937406    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.747903926 podStartE2EDuration="8.937389756s" totalImagesPullingTime="4.16903512s" totalInitContainerRuntime="20.45071ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:18.935657666 +0000 UTC m=+70.902835484" watchObservedRunningTime="2026-09-17 15:31:18.937389756 +0000 UTC m=+70.904567574"
Sep 17 15:31:18.969703 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.969634    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf" podStartSLOduration=2.756795436 podStartE2EDuration="6.96959778s" totalImagesPullingTime="4.212802344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:18.969288063 +0000 UTC m=+70.936465888" watchObservedRunningTime="2026-09-17 15:31:18.96959778 +0000 UTC m=+70.936775599"
Sep 17 15:31:18.995142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:18.995076    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5" podStartSLOduration=1.7919656019999999 podStartE2EDuration="4.995054975s" totalImagesPullingTime="3.203089373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:18.994551876 +0000 UTC m=+70.961729695" watchObservedRunningTime="2026-09-17 15:31:18.995054975 +0000 UTC m=+70.962232794"
Sep 17 15:31:23.888905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.888868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kg4qt" event={"ID":"c2cef1e7-01db-4378-ac73-9efa6ba71ee5","Type":"ContainerStarted","Data":"cfac0a015d36c9c2b0c832cb899280ad2bd415951cfa63f1d45ded09513bbfcc"}
Sep 17 15:31:23.890398 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.890372    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j" event={"ID":"7ef6441c-05be-4893-9f4b-97e7a8936a3e","Type":"ContainerStarted","Data":"f9fd63fc28febc1821a35fc4308c4ae4a32edd76d96e767641bd168cb62a9d0a"}
Sep 17 15:31:23.890512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.890483    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:23.891988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.891961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-67sjn" event={"ID":"96aac0ea-441d-4cfb-b9ca-31dc9855bbaf","Type":"ContainerStarted","Data":"3e9ba1efda44d6f63477c32e43d4ebb8e54cfda49cf488e5c85e8a1a8d589742"}
Sep 17 15:31:23.892077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.892017    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:31:23.895755 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"a8b9ed7ce125a5f5228b3da79cf72a6c61d02d892175cb57a92f8889df4c76a3"}
Sep 17 15:31:23.895848 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895794    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"4dc5e5efc44cc879ecde80ca3e95814063801af5dd5912b055c6af09d89446b8"}
Sep 17 15:31:23.895848 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895836    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"f3d319351f152391bfb802330aebc28264257cc595a2631d3a8cc35a24b3ded2"}
Sep 17 15:31:23.896114 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"6af1e03d7f2ec679db6c7faf54d5f2d5105cfeeffb372d22203ef38be9ad1716"}
Sep 17 15:31:23.896114 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895912    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"25d31a4ae3a7b458ccc2a54b919d0482c1d45b6aafa0c0dc8d41d3ff176d93ac"}
Sep 17 15:31:23.896114 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.895944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5","Type":"ContainerStarted","Data":"ca1040012df4fe20a18ea9c72c2768b6988b962ea26c6786f65f11b0f711bd45"}
Sep 17 15:31:23.896223 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.896159    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j"
Sep 17 15:31:23.907220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.907169    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kg4qt" podStartSLOduration=68.348175737 podStartE2EDuration="1m15.907151964s" totalImagesPullingTime="7.558976227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:23.90557846 +0000 UTC m=+75.872756280" watchObservedRunningTime="2026-09-17 15:31:23.907151964 +0000 UTC m=+75.874329785"
Sep 17 15:31:23.921441 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.921404    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5v27j" podStartSLOduration=5.246483489 podStartE2EDuration="9.921395227s" totalImagesPullingTime="4.674911738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:23.920790422 +0000 UTC m=+75.887968240" watchObservedRunningTime="2026-09-17 15:31:23.921395227 +0000 UTC m=+75.888573044"
Sep 17 15:31:23.958226 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.956421    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.910876218 podStartE2EDuration="8.956402294s" totalImagesPullingTime="4.020488613s" totalInitContainerRuntime="25.037463ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:31:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:23.952624907 +0000 UTC m=+75.919802717" watchObservedRunningTime="2026-09-17 15:31:23.956402294 +0000 UTC m=+75.923580113"
Sep 17 15:31:23.972704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:23.972649    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-67sjn" podStartSLOduration=68.413010913 podStartE2EDuration="1m15.972632452s" totalImagesPullingTime="7.559621539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:31:23.971540266 +0000 UTC m=+75.938718098" watchObservedRunningTime="2026-09-17 15:31:23.972632452 +0000 UTC m=+75.939810264"
Sep 17 15:31:24.874214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:24.874186    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7567c9f887-rqvrf"
Sep 17 15:31:26.348471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:26.348439    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:31:31.093422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:31.093303    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:31:34.383309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:34.383273    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:34.383761 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:34.383334    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:54.388886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:54.388856    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:54.392712 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:54.392690    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57c57d476b-mccw5"
Sep 17 15:31:54.901750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:54.901721    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-67sjn"
Sep 17 15:31:56.111792 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.111734    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bcb6795d6-kxl7s" podUID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" containerName="console" containerID="cri-o://7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25" gracePeriod=15
Sep 17 15:31:56.384415 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.384359    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:56.556157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556129    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556168    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556197    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556232    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556290    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556337    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5g94n\" (UniqueName: \"kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n\") pod \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\" (UID: \"21f2bae8-c94f-4b3d-b2b6-e73f383766d0\") "
Sep 17 15:31:56.556710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556673    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config" (OuterVolumeSpecName: "console-config") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:31:56.556710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556688    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:31:56.556910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.556886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca" (OuterVolumeSpecName: "service-ca") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:31:56.558786 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.558758    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:31:56.559004 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.558979    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n" (OuterVolumeSpecName: "kube-api-access-5g94n") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "kube-api-access-5g94n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:31:56.559056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.558990    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "21f2bae8-c94f-4b3d-b2b6-e73f383766d0" (UID: "21f2bae8-c94f-4b3d-b2b6-e73f383766d0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:31:56.658115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658036    2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-oauth-config\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:56.658115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658073    2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:56.658115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658088    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g94n\" (UniqueName: \"kubernetes.io/projected/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-kube-api-access-5g94n\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:56.658115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658103    2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-oauth-serving-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:56.658115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658116    2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-config\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:56.658364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:56.658130    2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/21f2bae8-c94f-4b3d-b2b6-e73f383766d0-console-serving-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:31:57.036480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.036445    2564 generic.go:356] "Generic (PLEG): container finished" podID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" containerID="7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25" exitCode=0
Sep 17 15:31:57.036682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.036495    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcb6795d6-kxl7s"
Sep 17 15:31:57.036682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.036539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb6795d6-kxl7s" event={"ID":"21f2bae8-c94f-4b3d-b2b6-e73f383766d0","Type":"ContainerDied","Data":"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"}
Sep 17 15:31:57.036682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.036592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcb6795d6-kxl7s" event={"ID":"21f2bae8-c94f-4b3d-b2b6-e73f383766d0","Type":"ContainerDied","Data":"28375c35f0aec4ed9baeeea1ad34b3d43ce5f74d2fde4c69823ca6e2825dd844"}
Sep 17 15:31:57.036682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.036645    2564 scope.go:117] "RemoveContainer" containerID="7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"
Sep 17 15:31:57.045218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.045199    2564 scope.go:117] "RemoveContainer" containerID="7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"
Sep 17 15:31:57.045480 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:31:57.045461    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25\": container with ID starting with 7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25 not found: ID does not exist" containerID="7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"
Sep 17 15:31:57.045536 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.045504    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25"} err="failed to get container status \"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25\": rpc error: code = NotFound desc = could not find container \"7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25\": container with ID starting with 7596c0c27dd8c0091919d79f5eb56d3b1b1dc2189f4ea624a2c4645d74f63e25 not found: ID does not exist"
Sep 17 15:31:57.068819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.068789    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:31:57.074889 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:57.074865    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcb6795d6-kxl7s"]
Sep 17 15:31:58.522869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:31:58.522835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" path="/var/lib/kubelet/pods/21f2bae8-c94f-4b3d-b2b6-e73f383766d0/volumes"
Sep 17 15:32:16.348479 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:32:16.348436    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:32:16.376784 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:32:16.376759    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:32:17.129634 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:32:17.129594    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:34:00.130416 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.130379    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2b8bt"]
Sep 17 15:34:00.130890 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.130676    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" containerName="console"
Sep 17 15:34:00.130890 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.130688    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" containerName="console"
Sep 17 15:34:00.130890 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.130741    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="21f2bae8-c94f-4b3d-b2b6-e73f383766d0" containerName="console"
Sep 17 15:34:00.132831 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.132816    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.135259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.135235    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 15:34:00.135392 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.135303    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 15:34:00.135490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.135475    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-6h7t5\""
Sep 17 15:34:00.149293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.149272    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2b8bt"]
Sep 17 15:34:00.180241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.180215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj5h6\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-kube-api-access-dj5h6\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.180326 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.180260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-bound-sa-token\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.281532 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.281505    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj5h6\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-kube-api-access-dj5h6\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.281642 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.281547    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-bound-sa-token\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.291042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.291017    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-bound-sa-token\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.292184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.292164    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj5h6\" (UniqueName: \"kubernetes.io/projected/7b889ba6-4eee-46ac-b995-d84854399922-kube-api-access-dj5h6\") pod \"cert-manager-55b4d954d5-2b8bt\" (UID: \"7b889ba6-4eee-46ac-b995-d84854399922\") " pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.455388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.455300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-2b8bt"
Sep 17 15:34:00.732706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:00.732682    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-2b8bt"]
Sep 17 15:34:00.737581 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:00.737542    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b889ba6_4eee_46ac_b995_d84854399922.slice/crio-38e4f1cc87fde193cbc834b6ba017917a5b3a75dbd2e8ed9aaa45bbd9284235a WatchSource:0}: Error finding container 38e4f1cc87fde193cbc834b6ba017917a5b3a75dbd2e8ed9aaa45bbd9284235a: Status 404 returned error can't find the container with id 38e4f1cc87fde193cbc834b6ba017917a5b3a75dbd2e8ed9aaa45bbd9284235a
Sep 17 15:34:01.495219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:01.495180    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2b8bt" event={"ID":"7b889ba6-4eee-46ac-b995-d84854399922","Type":"ContainerStarted","Data":"38e4f1cc87fde193cbc834b6ba017917a5b3a75dbd2e8ed9aaa45bbd9284235a"}
Sep 17 15:34:03.820382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.820356    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"]
Sep 17 15:34:03.822629 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.822601    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:03.825021 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.825000    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:34:03.825534 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.825516    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-h8jr5\""
Sep 17 15:34:03.826152 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.826130    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 15:34:03.836957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.836938    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"]
Sep 17 15:34:03.910541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.910515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/797a030a-0f4b-419e-938e-f29965836e7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:03.910662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:03.910558    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2sc\" (UniqueName: \"kubernetes.io/projected/797a030a-0f4b-419e-938e-f29965836e7e-kube-api-access-gd2sc\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.010875 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.010854    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/797a030a-0f4b-419e-938e-f29965836e7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.010955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.010896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2sc\" (UniqueName: \"kubernetes.io/projected/797a030a-0f4b-419e-938e-f29965836e7e-kube-api-access-gd2sc\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.011272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.011227    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/797a030a-0f4b-419e-938e-f29965836e7e-tmp\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.018960 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.018940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2sc\" (UniqueName: \"kubernetes.io/projected/797a030a-0f4b-419e-938e-f29965836e7e-kube-api-access-gd2sc\") pod \"openshift-lws-operator-fd8ccff4c-9p67z\" (UID: \"797a030a-0f4b-419e-938e-f29965836e7e\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.131586 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.131566    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"
Sep 17 15:34:04.405369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.405268    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z"]
Sep 17 15:34:04.407948 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:04.407912    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod797a030a_0f4b_419e_938e_f29965836e7e.slice/crio-f1a116c045ceb7d3149454eff2c0b35c49dee4fcd463f6d99d9c7c54bc3e6bdb WatchSource:0}: Error finding container f1a116c045ceb7d3149454eff2c0b35c49dee4fcd463f6d99d9c7c54bc3e6bdb: Status 404 returned error can't find the container with id f1a116c045ceb7d3149454eff2c0b35c49dee4fcd463f6d99d9c7c54bc3e6bdb
Sep 17 15:34:04.507290 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.507255    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-2b8bt" event={"ID":"7b889ba6-4eee-46ac-b995-d84854399922","Type":"ContainerStarted","Data":"7347b5097da458d8f38f4a6bfb6ea721922d862590b8179fa50a2278cd50224a"}
Sep 17 15:34:04.508591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.508570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z" event={"ID":"797a030a-0f4b-419e-938e-f29965836e7e","Type":"ContainerStarted","Data":"f1a116c045ceb7d3149454eff2c0b35c49dee4fcd463f6d99d9c7c54bc3e6bdb"}
Sep 17 15:34:04.530720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:04.530679    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-2b8bt" podStartSLOduration=1.535522776 podStartE2EDuration="4.530664385s" totalImagesPullingTime="2.995141609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:04.529016466 +0000 UTC m=+236.496194282" watchObservedRunningTime="2026-09-17 15:34:04.530664385 +0000 UTC m=+236.497842202"
Sep 17 15:34:07.524304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:07.524264    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z" event={"ID":"797a030a-0f4b-419e-938e-f29965836e7e","Type":"ContainerStarted","Data":"449ae35ce58fd089ad5a42914cf6d202a63c6a0a172118de2777c88394ce2c1b"}
Sep 17 15:34:07.546890 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:07.546838    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-9p67z" podStartSLOduration=2.316000896 podStartE2EDuration="4.546823068s" totalImagesPullingTime="2.230822172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:07.545796603 +0000 UTC m=+239.512974422" watchObservedRunningTime="2026-09-17 15:34:07.546823068 +0000 UTC m=+239.514000887"
Sep 17 15:34:27.276314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.276279    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"]
Sep 17 15:34:27.357936 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.357903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"]
Sep 17 15:34:27.358094 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.358028    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.362933 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.362914    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 15:34:27.363058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.362974    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 15:34:27.363058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.363036    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 15:34:27.365884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.365869    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r95mn\""
Sep 17 15:34:27.366456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.366437    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 15:34:27.501946 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.501911    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.502144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.501960    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.502144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.502041    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.502144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.502078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2bf\" (UniqueName: \"kubernetes.io/projected/77f44441-f9ee-43b6-ace4-f1432cf0b840-kube-api-access-6x2bf\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.502144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.502117    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.602950    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.602990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2bf\" (UniqueName: \"kubernetes.io/projected/77f44441-f9ee-43b6-ace4-f1432cf0b840-kube-api-access-6x2bf\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603252 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.603050    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603252 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.603075    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603252 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.603123    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.603429 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.603333    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-manifests\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.605438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.605404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/77f44441-f9ee-43b6-ace4-f1432cf0b840-tmp\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.605575 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.605556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-webhook-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.605657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.605602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/77f44441-f9ee-43b6-ace4-f1432cf0b840-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.611407 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.611388    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2bf\" (UniqueName: \"kubernetes.io/projected/77f44441-f9ee-43b6-ace4-f1432cf0b840-kube-api-access-6x2bf\") pod \"opendatahub-operator-controller-manager-5df9996869-ljl7k\" (UID: \"77f44441-f9ee-43b6-ace4-f1432cf0b840\") " pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.668422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.668387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:27.983574 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:27.983550    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"]
Sep 17 15:34:27.986301 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:27.986275    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f44441_f9ee_43b6_ace4_f1432cf0b840.slice/crio-c181e8385c980d5a22a80f82ae510affddf0175e874bd33158eb7e57059533a9 WatchSource:0}: Error finding container c181e8385c980d5a22a80f82ae510affddf0175e874bd33158eb7e57059533a9: Status 404 returned error can't find the container with id c181e8385c980d5a22a80f82ae510affddf0175e874bd33158eb7e57059533a9
Sep 17 15:34:28.598948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:28.598914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k" event={"ID":"77f44441-f9ee-43b6-ace4-f1432cf0b840","Type":"ContainerStarted","Data":"c181e8385c980d5a22a80f82ae510affddf0175e874bd33158eb7e57059533a9"}
Sep 17 15:34:31.613577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:31.613537    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k" event={"ID":"77f44441-f9ee-43b6-ace4-f1432cf0b840","Type":"ContainerStarted","Data":"20f5ac22f0bec2e89c41751f061a5dc69dfe69021b3b38959dfbb1b585539016"}
Sep 17 15:34:32.619702 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:32.619673    2564 generic.go:356] "Generic (PLEG): container finished" podID="77f44441-f9ee-43b6-ace4-f1432cf0b840" containerID="20f5ac22f0bec2e89c41751f061a5dc69dfe69021b3b38959dfbb1b585539016" exitCode=0
Sep 17 15:34:32.620091 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:32.619741    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k" event={"ID":"77f44441-f9ee-43b6-ace4-f1432cf0b840","Type":"ContainerDied","Data":"20f5ac22f0bec2e89c41751f061a5dc69dfe69021b3b38959dfbb1b585539016"}
Sep 17 15:34:33.625628 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:33.625580    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k" event={"ID":"77f44441-f9ee-43b6-ace4-f1432cf0b840","Type":"ContainerStarted","Data":"7e35ad4ad066353b736128ddd0f75754e55acdaae51e365a7318ba0bbaa02262"}
Sep 17 15:34:33.626002 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:33.625676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:33.654666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:33.654608    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k" podStartSLOduration=2.120785146 podStartE2EDuration="6.654592227s" totalImagesPullingTime="3.50669433s" totalInitContainerRuntime="1.027112751s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:33.65236087 +0000 UTC m=+265.619538688" watchObservedRunningTime="2026-09-17 15:34:33.654592227 +0000 UTC m=+265.621770046"
Sep 17 15:34:44.631519 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:44.631486    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5df9996869-ljl7k"
Sep 17 15:34:48.044806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.044769    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"]
Sep 17 15:34:48.050265 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.050244    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.057799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.057779    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 15:34:48.058013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.057994    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:34:48.058575 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.058561    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:34:48.058768 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.058754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 15:34:48.058873 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.058857    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6l7xx\""
Sep 17 15:34:48.061487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.061466    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2znhr\" (UniqueName: \"kubernetes.io/projected/baad6870-412b-4dc3-a310-87f370489f3e-kube-api-access-2znhr\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.061581 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.061493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baad6870-412b-4dc3-a310-87f370489f3e-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.061581 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.061510    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baad6870-412b-4dc3-a310-87f370489f3e-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.080073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.080055    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"]
Sep 17 15:34:48.162093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.162063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baad6870-412b-4dc3-a310-87f370489f3e-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.162227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.162166    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2znhr\" (UniqueName: \"kubernetes.io/projected/baad6870-412b-4dc3-a310-87f370489f3e-kube-api-access-2znhr\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.162227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.162198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baad6870-412b-4dc3-a310-87f370489f3e-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.164259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.164228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baad6870-412b-4dc3-a310-87f370489f3e-tmp\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.164492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.164475    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baad6870-412b-4dc3-a310-87f370489f3e-tls-certs\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.174859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.174839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2znhr\" (UniqueName: \"kubernetes.io/projected/baad6870-412b-4dc3-a310-87f370489f3e-kube-api-access-2znhr\") pod \"kube-auth-proxy-5dc8ddcd98-hvz5n\" (UID: \"baad6870-412b-4dc3-a310-87f370489f3e\") " pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.360268 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.360202    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"
Sep 17 15:34:48.631705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.631638    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n"]
Sep 17 15:34:48.635005 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:48.634969    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaad6870_412b_4dc3_a310_87f370489f3e.slice/crio-9c57cb1711ce1b5d06ed658c773a918fe347eeda48cbcc7bba6e16a2c244720b WatchSource:0}: Error finding container 9c57cb1711ce1b5d06ed658c773a918fe347eeda48cbcc7bba6e16a2c244720b: Status 404 returned error can't find the container with id 9c57cb1711ce1b5d06ed658c773a918fe347eeda48cbcc7bba6e16a2c244720b
Sep 17 15:34:48.687411 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:48.687384    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n" event={"ID":"baad6870-412b-4dc3-a310-87f370489f3e","Type":"ContainerStarted","Data":"9c57cb1711ce1b5d06ed658c773a918fe347eeda48cbcc7bba6e16a2c244720b"}
Sep 17 15:34:51.700917 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:51.700832    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n" event={"ID":"baad6870-412b-4dc3-a310-87f370489f3e","Type":"ContainerStarted","Data":"ca0b2420a63500f06a35323c389aaac295942db0c771100c38efc58e7adbfa51"}
Sep 17 15:34:51.722814 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:51.722768    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5dc8ddcd98-hvz5n" podStartSLOduration=0.972111446 podStartE2EDuration="3.722752963s" totalImagesPullingTime="2.750641517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:34:51.721400704 +0000 UTC m=+283.688578521" watchObservedRunningTime="2026-09-17 15:34:51.722752963 +0000 UTC m=+283.689930780"
Sep 17 15:34:54.252966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.252933    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"]
Sep 17 15:34:54.255193 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.255172    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.257201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.257179    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-vnggk\""
Sep 17 15:34:54.257309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.257220    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 15:34:54.269486 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.269464    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"]
Sep 17 15:34:54.314073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314048    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314079    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314112    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314239 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314239 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314215    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zs9b\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-kube-api-access-7zs9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314278    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.314394 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.314325    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415201    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415251    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415289    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415330    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415359    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zs9b\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-kube-api-access-7zs9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415393    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415496    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415679    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415806    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.415904 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.415880    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.416132 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.416107    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.416219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.416199    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.418093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.418073    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.418297 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.418281    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.424198 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.424165    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zs9b\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-kube-api-access-7zs9b\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.424551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.424530    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4daa9b53-6d59-4199-8d2a-739b7b2306c8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz\" (UID: \"4daa9b53-6d59-4199-8d2a-739b7b2306c8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.566563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.566532    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:34:54.837292 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:54.837256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"]
Sep 17 15:34:54.839464 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:54.839431    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4daa9b53_6d59_4199_8d2a_739b7b2306c8.slice/crio-17e6d5ff53780e7bc224ecaa32c37839397aa2f9a74bc9eccbedc2737f64345e WatchSource:0}: Error finding container 17e6d5ff53780e7bc224ecaa32c37839397aa2f9a74bc9eccbedc2737f64345e: Status 404 returned error can't find the container with id 17e6d5ff53780e7bc224ecaa32c37839397aa2f9a74bc9eccbedc2737f64345e
Sep 17 15:34:55.720656 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:55.720606    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz" event={"ID":"4daa9b53-6d59-4199-8d2a-739b7b2306c8","Type":"ContainerStarted","Data":"17e6d5ff53780e7bc224ecaa32c37839397aa2f9a74bc9eccbedc2737f64345e"}
Sep 17 15:34:58.774097 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.774065    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ctnnt"]
Sep 17 15:34:58.776877 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.776861    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:58.779625 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.779582    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-c5dzc\""
Sep 17 15:34:58.779769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.779754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Sep 17 15:34:58.801218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.801197    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ctnnt"]
Sep 17 15:34:58.852432 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.852410    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:58.852535 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.852462    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx24d\" (UniqueName: \"kubernetes.io/projected/86d1a699-cf68-4442-b8d1-1b1c85c32726-kube-api-access-cx24d\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:58.953377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.953345    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx24d\" (UniqueName: \"kubernetes.io/projected/86d1a699-cf68-4442-b8d1-1b1c85c32726-kube-api-access-cx24d\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:58.953500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.953410    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:58.953564 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:34:58.953545    2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Sep 17 15:34:58.953648 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:34:58.953597    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert podName:86d1a699-cf68-4442-b8d1-1b1c85c32726 nodeName:}" failed. No retries permitted until 2026-09-17 15:34:59.453581401 +0000 UTC m=+291.420759197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert") pod "odh-model-controller-7998c67dc4-ctnnt" (UID: "86d1a699-cf68-4442-b8d1-1b1c85c32726") : secret "odh-model-controller-webhook-cert" not found
Sep 17 15:34:58.963666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:58.963644    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx24d\" (UniqueName: \"kubernetes.io/projected/86d1a699-cf68-4442-b8d1-1b1c85c32726-kube-api-access-cx24d\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:59.456657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:59.456598    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:59.458803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:59.458780    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/86d1a699-cf68-4442-b8d1-1b1c85c32726-cert\") pod \"odh-model-controller-7998c67dc4-ctnnt\" (UID: \"86d1a699-cf68-4442-b8d1-1b1c85c32726\") " pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:59.686844 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:59.686810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:34:59.954087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:34:59.954057    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ctnnt"]
Sep 17 15:34:59.957609 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:34:59.957581    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d1a699_cf68_4442_b8d1_1b1c85c32726.slice/crio-59ec9e5ccdead196dda2595c17acde05d5e4b3e8969639fbd10fd4c2ea415230 WatchSource:0}: Error finding container 59ec9e5ccdead196dda2595c17acde05d5e4b3e8969639fbd10fd4c2ea415230: Status 404 returned error can't find the container with id 59ec9e5ccdead196dda2595c17acde05d5e4b3e8969639fbd10fd4c2ea415230
Sep 17 15:35:00.744089 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:00.744048    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt" event={"ID":"86d1a699-cf68-4442-b8d1-1b1c85c32726","Type":"ContainerStarted","Data":"59ec9e5ccdead196dda2595c17acde05d5e4b3e8969639fbd10fd4c2ea415230"}
Sep 17 15:35:03.759341 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:03.759299    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt" event={"ID":"86d1a699-cf68-4442-b8d1-1b1c85c32726","Type":"ContainerStarted","Data":"17e082da21028d44332033e2ac4533a2a311cb58e09265b4145185ca41fda600"}
Sep 17 15:35:03.759779 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:03.759424    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:35:03.778183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:03.778142    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt" podStartSLOduration=3.025017992 podStartE2EDuration="5.778128184s" totalImagesPullingTime="2.753110192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:03.777553696 +0000 UTC m=+295.744731515" watchObservedRunningTime="2026-09-17 15:35:03.778128184 +0000 UTC m=+295.745306001"
Sep 17 15:35:05.598877 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.598849    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"]
Sep 17 15:35:05.675833 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.675797    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"]
Sep 17 15:35:05.675985 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.675916    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.680427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.680410    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 17 15:35:05.680818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.680803    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 17 15:35:05.680818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.680812    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 17 15:35:05.681051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.681034    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-2964l\""
Sep 17 15:35:05.812638 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812584    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812671    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812709    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af846e86-77ae-4292-aaf4-38824b3dfe77-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812735    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbq8\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-kube-api-access-2gbq8\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812974 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812855    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812974 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812900    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.812974 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.812929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914061    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914365 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914182    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914365 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af846e86-77ae-4292-aaf4-38824b3dfe77-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914365 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914257    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbq8\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-kube-api-access-2gbq8\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914365 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914331    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.914662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.914382    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.915416 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.915380    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.916838 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.916810    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af846e86-77ae-4292-aaf4-38824b3dfe77-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.916954 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.916885    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.917148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.917125    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.917187 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.917163    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.922771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.922744    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.923147 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.923117    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbq8\" (UniqueName: \"kubernetes.io/projected/af846e86-77ae-4292-aaf4-38824b3dfe77-kube-api-access-2gbq8\") pod \"istiod-openshift-gateway-5677ccb49b-gv5kp\" (UID: \"af846e86-77ae-4292-aaf4-38824b3dfe77\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:05.984886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:05.984847    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:06.290216 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:06.290193    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"]
Sep 17 15:35:06.291908 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:35:06.291883    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf846e86_77ae_4292_aaf4_38824b3dfe77.slice/crio-acf23f5ed30cae22be4de56758ee8bbf217ed369faa6139f30de6ec750ce8f1b WatchSource:0}: Error finding container acf23f5ed30cae22be4de56758ee8bbf217ed369faa6139f30de6ec750ce8f1b: Status 404 returned error can't find the container with id acf23f5ed30cae22be4de56758ee8bbf217ed369faa6139f30de6ec750ce8f1b
Sep 17 15:35:06.779800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:06.779769    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp" event={"ID":"af846e86-77ae-4292-aaf4-38824b3dfe77","Type":"ContainerStarted","Data":"acf23f5ed30cae22be4de56758ee8bbf217ed369faa6139f30de6ec750ce8f1b"}
Sep 17 15:35:08.693458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.693426    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"]
Sep 17 15:35:08.695654 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.695638    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:08.697884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.697861    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fjjqf\""
Sep 17 15:35:08.698249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.698232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 17 15:35:08.715581 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.715555    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"]
Sep 17 15:35:08.841502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.841470    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdhqx\" (UniqueName: \"kubernetes.io/projected/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-kube-api-access-jdhqx\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:08.841690 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.841626    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:08.942783 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.942742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:08.942971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.942799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdhqx\" (UniqueName: \"kubernetes.io/projected/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-kube-api-access-jdhqx\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:08.942971 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:35:08.942915    2564 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 15:35:08.943090 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:35:08.942993    2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert podName:0b3630db-aaf8-4afa-9b5b-1676376ed7f7 nodeName:}" failed. No retries permitted until 2026-09-17 15:35:09.442970723 +0000 UTC m=+301.410148521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert") pod "kserve-controller-manager-d6f4b65b8-vqx65" (UID: "0b3630db-aaf8-4afa-9b5b-1676376ed7f7") : secret "kserve-webhook-server-cert" not found
Sep 17 15:35:08.957094 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:08.957027    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdhqx\" (UniqueName: \"kubernetes.io/projected/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-kube-api-access-jdhqx\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:09.447086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.447043    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:09.449908 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.449878    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b3630db-aaf8-4afa-9b5b-1676376ed7f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-vqx65\" (UID: \"0b3630db-aaf8-4afa-9b5b-1676376ed7f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:09.605292 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.605256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:09.682561 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.682512    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:35:09.682679 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.682597    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:35:09.690875 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.690840    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:35:09.690972 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.690930    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:35:09.691022 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.690972    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:35:09.717919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.717881    2564 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:35:09.798025 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.797551    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz" event={"ID":"4daa9b53-6d59-4199-8d2a-739b7b2306c8","Type":"ContainerStarted","Data":"9fbfcdb6da3f3299e7b21940a7e9f229418ec0b81e4913cf20043ae8185b4836"}
Sep 17 15:35:09.800951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.800906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp" event={"ID":"af846e86-77ae-4292-aaf4-38824b3dfe77","Type":"ContainerStarted","Data":"b242036081a6b9dd89161f5faef9f7aa22589aa8724cb645dfc8692f02c2c72b"}
Sep 17 15:35:09.801230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.801179    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:09.834385 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.834317    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz" podStartSLOduration=0.98571044 podStartE2EDuration="15.834297227s" totalImagesPullingTime="14.848586787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:09.830825774 +0000 UTC m=+301.798003591" watchObservedRunningTime="2026-09-17 15:35:09.834297227 +0000 UTC m=+301.801475046"
Sep 17 15:35:09.891741 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.891689    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp" podStartSLOduration=1.503626213 podStartE2EDuration="4.891672487s" totalImagesPullingTime="3.388046274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:09.889426676 +0000 UTC m=+301.856604496" watchObservedRunningTime="2026-09-17 15:35:09.891672487 +0000 UTC m=+301.858850306"
Sep 17 15:35:09.985953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.985926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"]
Sep 17 15:35:09.988803 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:35:09.988775    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3630db_aaf8_4afa_9b5b_1676376ed7f7.slice/crio-d1d943f85da79504b531613ba27d5ed402b17bc715969186bd7a9dfb0d1f863f WatchSource:0}: Error finding container d1d943f85da79504b531613ba27d5ed402b17bc715969186bd7a9dfb0d1f863f: Status 404 returned error can't find the container with id d1d943f85da79504b531613ba27d5ed402b17bc715969186bd7a9dfb0d1f863f
Sep 17 15:35:09.990373 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:09.990356    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:35:10.567493 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:10.567464    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:35:10.569062 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:10.569034    2564 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused" start-of-body=
Sep 17 15:35:10.569174 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:10.569086    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz" podUID="4daa9b53-6d59-4199-8d2a-739b7b2306c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.20:15021/healthz/ready\": dial tcp 10.133.0.20:15021: connect: connection refused"
Sep 17 15:35:10.807327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:10.807285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65" event={"ID":"0b3630db-aaf8-4afa-9b5b-1676376ed7f7","Type":"ContainerStarted","Data":"d1d943f85da79504b531613ba27d5ed402b17bc715969186bd7a9dfb0d1f863f"}
Sep 17 15:35:10.809477 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:10.809455    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-gv5kp"
Sep 17 15:35:11.571018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:11.570993    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:35:11.811240 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:11.811215    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:35:11.812227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:11.812206    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz"
Sep 17 15:35:12.817125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:12.817084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65" event={"ID":"0b3630db-aaf8-4afa-9b5b-1676376ed7f7","Type":"ContainerStarted","Data":"3b2ab81c693b3425602bfdb71a330227619d209d689f86d2f23eb61da57acac5"}
Sep 17 15:35:12.817563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:12.817171    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:35:12.836680 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:12.836596    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65" podStartSLOduration=2.226962185 podStartE2EDuration="4.836580314s" totalImagesPullingTime="2.609618129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:35:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:35:12.835595704 +0000 UTC m=+304.802773522" watchObservedRunningTime="2026-09-17 15:35:12.836580314 +0000 UTC m=+304.803758132"
Sep 17 15:35:14.766073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:14.766042    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ctnnt"
Sep 17 15:35:43.827394 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:35:43.827366    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-vqx65"
Sep 17 15:36:07.567424 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.567393    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-978784f69-msg4d"]
Sep 17 15:36:07.569670 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.569654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.576446 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.576429    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 15:36:07.576543 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.576430    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 15:36:07.577084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.577066    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 15:36:07.577259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.577238    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 15:36:07.577874 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.577858    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 15:36:07.578109 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.578090    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 15:36:07.578323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.578306    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 15:36:07.578822 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.578802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-4tjvv\""
Sep 17 15:36:07.582473 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.582455    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 15:36:07.588652 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.588608    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978784f69-msg4d"]
Sep 17 15:36:07.627961 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.627939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-oauth-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.627974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.627989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56w2v\" (UniqueName: \"kubernetes.io/projected/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-kube-api-access-56w2v\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.628085    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-trusted-ca-bundle\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.628102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628182 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.628172    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-service-ca\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.628214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.628191    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-oauth-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729610 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729573    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-oauth-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729638    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729659    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56w2v\" (UniqueName: \"kubernetes.io/projected/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-kube-api-access-56w2v\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729702    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-trusted-ca-bundle\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729821    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-service-ca\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.729967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.729843    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-oauth-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.730474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.730451    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.730574 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.730537    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-trusted-ca-bundle\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.730654 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.730609    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-service-ca\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.730744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.730724    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-oauth-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.732091 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.732069    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-oauth-config\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.732245 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.732230    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-console-serving-cert\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.740404 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.740383    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56w2v\" (UniqueName: \"kubernetes.io/projected/e85bdbdc-88fe-4014-bd9e-ced865bb69e7-kube-api-access-56w2v\") pod \"console-978784f69-msg4d\" (UID: \"e85bdbdc-88fe-4014-bd9e-ced865bb69e7\") " pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:07.879162 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:07.879106    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:08.160290 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:08.160219    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-978784f69-msg4d"]
Sep 17 15:36:08.163229 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:36:08.163198    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode85bdbdc_88fe_4014_bd9e_ced865bb69e7.slice/crio-0c6124e2282fed70e3f81d655ebc10d05f943010e820341649066b78ba49e6ea WatchSource:0}: Error finding container 0c6124e2282fed70e3f81d655ebc10d05f943010e820341649066b78ba49e6ea: Status 404 returned error can't find the container with id 0c6124e2282fed70e3f81d655ebc10d05f943010e820341649066b78ba49e6ea
Sep 17 15:36:09.069399 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:09.069361    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978784f69-msg4d" event={"ID":"e85bdbdc-88fe-4014-bd9e-ced865bb69e7","Type":"ContainerStarted","Data":"c45748cb4f25960387483f9f80bc3dff3b7250228270650f8845779631cd911a"}
Sep 17 15:36:09.069848 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:09.069440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-978784f69-msg4d" event={"ID":"e85bdbdc-88fe-4014-bd9e-ced865bb69e7","Type":"ContainerStarted","Data":"0c6124e2282fed70e3f81d655ebc10d05f943010e820341649066b78ba49e6ea"}
Sep 17 15:36:09.091793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:09.091741    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-978784f69-msg4d" podStartSLOduration=2.091725753 podStartE2EDuration="2.091725753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:09.089281488 +0000 UTC m=+361.056459305" watchObservedRunningTime="2026-09-17 15:36:09.091725753 +0000 UTC m=+361.058903571"
Sep 17 15:36:17.879375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:17.879336    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:17.879825 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:17.879436    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:17.884331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:17.884308    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:18.114106 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:18.114076    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-978784f69-msg4d"
Sep 17 15:36:26.843580 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.843540    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"]
Sep 17 15:36:26.846169 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.846151    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:26.848828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.848809    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-j9sqn\""
Sep 17 15:36:26.849234 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.849211    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 15:36:26.849351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.849239    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 15:36:26.879448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:26.879415    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"]
Sep 17 15:36:27.008183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:27.008144    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9frh\" (UniqueName: \"kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh\") pod \"limitador-operator-controller-manager-74495c69f8-gmw52\" (UID: \"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:27.109503 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:27.109412    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9frh\" (UniqueName: \"kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh\") pod \"limitador-operator-controller-manager-74495c69f8-gmw52\" (UID: \"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:27.125356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:27.125324    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9frh\" (UniqueName: \"kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh\") pod \"limitador-operator-controller-manager-74495c69f8-gmw52\" (UID: \"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:27.156469 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:27.156439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:27.463525 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:27.463501    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"]
Sep 17 15:36:27.465117 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:36:27.465090    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e2a3ae_9c42_41a9_a719_ef06d5ee14c0.slice/crio-43a030523ae25b8dcf618478addcefeddf1e9d3704e9ab63b219a508e9e01591 WatchSource:0}: Error finding container 43a030523ae25b8dcf618478addcefeddf1e9d3704e9ab63b219a508e9e01591: Status 404 returned error can't find the container with id 43a030523ae25b8dcf618478addcefeddf1e9d3704e9ab63b219a508e9e01591
Sep 17 15:36:28.153332 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:28.153298    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52" event={"ID":"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0","Type":"ContainerStarted","Data":"43a030523ae25b8dcf618478addcefeddf1e9d3704e9ab63b219a508e9e01591"}
Sep 17 15:36:30.163665 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:30.163609    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52" event={"ID":"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0","Type":"ContainerStarted","Data":"e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d"}
Sep 17 15:36:30.164040 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:30.163688    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:30.230879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:30.230827    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52" podStartSLOduration=1.782723744 podStartE2EDuration="4.230810389s" totalImagesPullingTime="2.448086645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:30.230033336 +0000 UTC m=+382.197211155" watchObservedRunningTime="2026-09-17 15:36:30.230810389 +0000 UTC m=+382.197988205"
Sep 17 15:36:33.078694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.078666    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"]
Sep 17 15:36:33.080883 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.080868    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.095104 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.095083    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j9zv8\""
Sep 17 15:36:33.123456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.123433    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"]
Sep 17 15:36:33.159669 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.159640    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tch8v\" (UniqueName: \"kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.159788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.159687    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.260607 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.260582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.260738 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.260692    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tch8v\" (UniqueName: \"kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.260956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.260938    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.279540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.279522    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tch8v\" (UniqueName: \"kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.390478 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.390421    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:33.732103 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:33.732029    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"]
Sep 17 15:36:33.744368 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:36:33.744342    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d26cc9_5509_45d0_b4a3_75bda4677c70.slice/crio-9211e5460ce64f67af4971d554319be243d81c07f90c9b3aa065a9c1cd2355bd WatchSource:0}: Error finding container 9211e5460ce64f67af4971d554319be243d81c07f90c9b3aa065a9c1cd2355bd: Status 404 returned error can't find the container with id 9211e5460ce64f67af4971d554319be243d81c07f90c9b3aa065a9c1cd2355bd
Sep 17 15:36:34.187281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:34.187250    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms" event={"ID":"66d26cc9-5509-45d0-b4a3-75bda4677c70","Type":"ContainerStarted","Data":"9211e5460ce64f67af4971d554319be243d81c07f90c9b3aa065a9c1cd2355bd"}
Sep 17 15:36:38.213179 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:38.213140    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms" event={"ID":"66d26cc9-5509-45d0-b4a3-75bda4677c70","Type":"ContainerStarted","Data":"9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644"}
Sep 17 15:36:38.213549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:38.213250    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:38.272332 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:38.272278    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms" podStartSLOduration=1.6387213040000002 podStartE2EDuration="5.272261612s" totalImagesPullingTime="3.633540308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:38.269493164 +0000 UTC m=+390.236670983" watchObservedRunningTime="2026-09-17 15:36:38.272261612 +0000 UTC m=+390.239439430"
Sep 17 15:36:41.175592 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:41.175551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:49.221843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:49.221810    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:52.679661 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.679625    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"]
Sep 17 15:36:52.680093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.679802    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" containerName="manager" containerID="cri-o://9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644" gracePeriod=2
Sep 17 15:36:52.693835 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.693813    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"]
Sep 17 15:36:52.725061 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.725036    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"]
Sep 17 15:36:52.725291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.725234    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" containerName="manager" containerID="cri-o://e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d" gracePeriod=2
Sep 17 15:36:52.733153 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:52.733118    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:52.753422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.753395    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"]
Sep 17 15:36:52.801302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801273    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:36:52.801832 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801813    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" containerName="manager"
Sep 17 15:36:52.801892 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801837    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" containerName="manager"
Sep 17 15:36:52.801892 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801867    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" containerName="manager"
Sep 17 15:36:52.801892 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801876    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" containerName="manager"
Sep 17 15:36:52.801982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" containerName="manager"
Sep 17 15:36:52.801982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.801976    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" containerName="manager"
Sep 17 15:36:52.804157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.804137    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:52.807979 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:52.807946    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:52.816139 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:52.816108    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:52.852849 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.852804    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"]
Sep 17 15:36:52.855737 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.855710    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:52.857222 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.857196    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:36:52.878710 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.878684    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"]
Sep 17 15:36:52.906597 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:52.906559    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:52.935001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.934974    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:52.935085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.935024    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc9tk\" (UniqueName: \"kubernetes.io/projected/3fe3e007-296a-4f1e-8a88-1467d2f0c8cf-kube-api-access-gc9tk\") pod \"limitador-operator-controller-manager-74495c69f8-dd6ms\" (UID: \"3fe3e007-296a-4f1e-8a88-1467d2f0c8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:52.935128 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.935084    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc9rn\" (UniqueName: \"kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:52.945844 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:52.945809    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:52.953363 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.953344    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:52.979818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:52.979801    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:53.020485 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.020461    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:53.022317 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.022300    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:53.035658 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035639    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tch8v\" (UniqueName: \"kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v\") pod \"66d26cc9-5509-45d0-b4a3-75bda4677c70\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") "
Sep 17 15:36:53.035737 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035697    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume\") pod \"66d26cc9-5509-45d0-b4a3-75bda4677c70\" (UID: \"66d26cc9-5509-45d0-b4a3-75bda4677c70\") "
Sep 17 15:36:53.035737 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035722    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9frh\" (UniqueName: \"kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh\") pod \"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0\" (UID: \"69e2a3ae-9c42-41a9-a719-ef06d5ee14c0\") "
Sep 17 15:36:53.035896 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035877    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:53.035944 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035928    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc9tk\" (UniqueName: \"kubernetes.io/projected/3fe3e007-296a-4f1e-8a88-1467d2f0c8cf-kube-api-access-gc9tk\") pod \"limitador-operator-controller-manager-74495c69f8-dd6ms\" (UID: \"3fe3e007-296a-4f1e-8a88-1467d2f0c8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:53.036005 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.035990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc9rn\" (UniqueName: \"kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:53.036310 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.036277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "66d26cc9-5509-45d0-b4a3-75bda4677c70" (UID: "66d26cc9-5509-45d0-b4a3-75bda4677c70"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:36:53.036310 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.036294    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:53.037833 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.037814    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v" (OuterVolumeSpecName: "kube-api-access-tch8v") pod "66d26cc9-5509-45d0-b4a3-75bda4677c70" (UID: "66d26cc9-5509-45d0-b4a3-75bda4677c70"). InnerVolumeSpecName "kube-api-access-tch8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:36:53.037897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.037879    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh" (OuterVolumeSpecName: "kube-api-access-f9frh") pod "69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" (UID: "69e2a3ae-9c42-41a9-a719-ef06d5ee14c0"). InnerVolumeSpecName "kube-api-access-f9frh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:36:53.081177 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.081154    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc9tk\" (UniqueName: \"kubernetes.io/projected/3fe3e007-296a-4f1e-8a88-1467d2f0c8cf-kube-api-access-gc9tk\") pod \"limitador-operator-controller-manager-74495c69f8-dd6ms\" (UID: \"3fe3e007-296a-4f1e-8a88-1467d2f0c8cf\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:53.136372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.136348    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tch8v\" (UniqueName: \"kubernetes.io/projected/66d26cc9-5509-45d0-b4a3-75bda4677c70-kube-api-access-tch8v\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:36:53.136372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.136366    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/66d26cc9-5509-45d0-b4a3-75bda4677c70-extensions-socket-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:36:53.136484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.136377    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9frh\" (UniqueName: \"kubernetes.io/projected/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0-kube-api-access-f9frh\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:36:53.138012 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.137995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc9rn\" (UniqueName: \"kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qw4gb\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:53.171190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.171168    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:53.286757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.286723    2564 generic.go:356] "Generic (PLEG): container finished" podID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" containerID="e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d" exitCode=0
Sep 17 15:36:53.286891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.286770    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:53.286891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.286825    2564 scope.go:117] "RemoveContainer" containerID="e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d"
Sep 17 15:36:53.288253 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.288227    2564 generic.go:356] "Generic (PLEG): container finished" podID="66d26cc9-5509-45d0-b4a3-75bda4677c70" containerID="9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644" exitCode=0
Sep 17 15:36:53.288366 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.288286    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:53.295399 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.295384    2564 scope.go:117] "RemoveContainer" containerID="e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d"
Sep 17 15:36:53.295787 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.295670    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d\": container with ID starting with e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d not found: ID does not exist" containerID="e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d"
Sep 17 15:36:53.295787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.295706    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d"} err="failed to get container status \"e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d\": rpc error: code = NotFound desc = could not find container \"e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d\": container with ID starting with e22a17706a697766f8f8edaea64d0b18fe7cea34d140e902532e605e6eb15a6d not found: ID does not exist"
Sep 17 15:36:53.295787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.295730    2564 scope.go:117] "RemoveContainer" containerID="9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644"
Sep 17 15:36:53.303058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.303042    2564 scope.go:117] "RemoveContainer" containerID="9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644"
Sep 17 15:36:53.303299 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.303282    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644\": container with ID starting with 9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644 not found: ID does not exist" containerID="9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644"
Sep 17 15:36:53.303343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.303305    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644"} err="failed to get container status \"9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644\": rpc error: code = NotFound desc = could not find container \"9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644\": container with ID starting with 9c8ba4505113eba3e786566525c8f2d2b7f833561c8fc03253f77a0eb3f77644 not found: ID does not exist"
Sep 17 15:36:53.315438 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.315415    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:53.317838 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.317819    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:53.322714 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.322696    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:53.324815 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:53.324796    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:53.413787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.413767    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:53.492811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.492751    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"]
Sep 17 15:36:53.504537 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:36:53.504514    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fe3e007_296a_4f1e_8a88_1467d2f0c8cf.slice/crio-110a2413f5335c53ee9e1b4b261c470d3a343df23237e92bfd0a2112ab0a303f WatchSource:0}: Error finding container 110a2413f5335c53ee9e1b4b261c470d3a343df23237e92bfd0a2112ab0a303f: Status 404 returned error can't find the container with id 110a2413f5335c53ee9e1b4b261c470d3a343df23237e92bfd0a2112ab0a303f
Sep 17 15:36:53.822438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:53.818112    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:36:53.824275 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:36:53.824249    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fb008fd_a61b_4e19_9c78_5dbf2acb5bec.slice/crio-b23512b2c75994a975f47e86b51340961cc352eeceb8e9587c30664ac382b549 WatchSource:0}: Error finding container b23512b2c75994a975f47e86b51340961cc352eeceb8e9587c30664ac382b549: Status 404 returned error can't find the container with id b23512b2c75994a975f47e86b51340961cc352eeceb8e9587c30664ac382b549
Sep 17 15:36:54.294629 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.294591    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" event={"ID":"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec","Type":"ContainerStarted","Data":"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"}
Sep 17 15:36:54.294869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.294671    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" event={"ID":"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec","Type":"ContainerStarted","Data":"b23512b2c75994a975f47e86b51340961cc352eeceb8e9587c30664ac382b549"}
Sep 17 15:36:54.294869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.294792    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:36:54.298133 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:54.298097    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:54.299017 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.298992    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms" event={"ID":"3fe3e007-296a-4f1e-8a88-1467d2f0c8cf","Type":"ContainerStarted","Data":"015bbd55fe9ef6591c79cffaf21e6ee23f0178e70fee59da04072dcd0b08825c"}
Sep 17 15:36:54.299149 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.299033    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms" event={"ID":"3fe3e007-296a-4f1e-8a88-1467d2f0c8cf","Type":"ContainerStarted","Data":"110a2413f5335c53ee9e1b4b261c470d3a343df23237e92bfd0a2112ab0a303f"}
Sep 17 15:36:54.299149 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.299062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:36:54.300250 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:54.300222    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:54.356970 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.356929    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" podStartSLOduration=2.356916073 podStartE2EDuration="2.356916073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:54.355699269 +0000 UTC m=+406.322877087" watchObservedRunningTime="2026-09-17 15:36:54.356916073 +0000 UTC m=+406.324093890"
Sep 17 15:36:54.406938 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.406888    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms" podStartSLOduration=2.406870891 podStartE2EDuration="2.406870891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:36:54.405123685 +0000 UTC m=+406.372301504" watchObservedRunningTime="2026-09-17 15:36:54.406870891 +0000 UTC m=+406.374048711"
Sep 17 15:36:54.408698 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:54.408675    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-s6xms\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s6xms"
Sep 17 15:36:54.415855 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:36:54.415827    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-gmw52\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-gmw52"
Sep 17 15:36:54.524221 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.524193    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d26cc9-5509-45d0-b4a3-75bda4677c70" path="/var/lib/kubelet/pods/66d26cc9-5509-45d0-b4a3-75bda4677c70/volumes"
Sep 17 15:36:54.524532 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:36:54.524518    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69e2a3ae-9c42-41a9-a719-ef06d5ee14c0" path="/var/lib/kubelet/pods/69e2a3ae-9c42-41a9-a719-ef06d5ee14c0/volumes"
Sep 17 15:37:00.679909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.679870    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 15:37:00.686468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.686442    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.706591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.706553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 15:37:00.809119 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.809078    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.809119 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.809120    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfwsf\" (UniqueName: \"kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.909756 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.909712    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.909937 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.909763    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfwsf\" (UniqueName: \"kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.910099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.910080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.933551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.933491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfwsf\" (UniqueName: \"kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-cvdr4\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:00.996300 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:00.996265    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:01.287321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:01.287293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 15:37:01.294886 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:37:01.294860    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod012d713f_2771_466f_9514_c5aba3196461.slice/crio-6aefab9d530f39519432278b6f2844339c3f6966881cc285779e62be83e668ca WatchSource:0}: Error finding container 6aefab9d530f39519432278b6f2844339c3f6966881cc285779e62be83e668ca: Status 404 returned error can't find the container with id 6aefab9d530f39519432278b6f2844339c3f6966881cc285779e62be83e668ca
Sep 17 15:37:01.336832 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:01.336805    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" event={"ID":"012d713f-2771-466f-9514-c5aba3196461","Type":"ContainerStarted","Data":"6aefab9d530f39519432278b6f2844339c3f6966881cc285779e62be83e668ca"}
Sep 17 15:37:02.343003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:02.342963    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" event={"ID":"012d713f-2771-466f-9514-c5aba3196461","Type":"ContainerStarted","Data":"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"}
Sep 17 15:37:02.343378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:02.343024    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:02.409941 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:02.409877    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" podStartSLOduration=2.409861054 podStartE2EDuration="2.409861054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:02.408975542 +0000 UTC m=+414.376153362" watchObservedRunningTime="2026-09-17 15:37:02.409861054 +0000 UTC m=+414.377038871"
Sep 17 15:37:05.308126 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:05.308093    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dd6ms"
Sep 17 15:37:05.308488 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:05.308204    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:37:13.349760 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.349732    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 15:37:13.460039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.460010    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:37:13.460215 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.460182    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" podUID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" containerName="manager" containerID="cri-o://3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e" gracePeriod=10
Sep 17 15:37:13.699206 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.699184    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:37:13.824996 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.824966    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume\") pod \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") "
Sep 17 15:37:13.825125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.825005    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc9rn\" (UniqueName: \"kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn\") pod \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\" (UID: \"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec\") "
Sep 17 15:37:13.825345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.825320    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" (UID: "2fb008fd-a61b-4e19-9c78-5dbf2acb5bec"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:37:13.827071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.827046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn" (OuterVolumeSpecName: "kube-api-access-qc9rn") pod "2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" (UID: "2fb008fd-a61b-4e19-9c78-5dbf2acb5bec"). InnerVolumeSpecName "kube-api-access-qc9rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:37:13.926221 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.926163    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-extensions-socket-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:37:13.926221 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:13.926184    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc9rn\" (UniqueName: \"kubernetes.io/projected/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec-kube-api-access-qc9rn\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:37:14.401372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.401342    2564 generic.go:356] "Generic (PLEG): container finished" podID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" containerID="3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e" exitCode=0
Sep 17 15:37:14.401757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.401388    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"
Sep 17 15:37:14.401757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.401415    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" event={"ID":"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec","Type":"ContainerDied","Data":"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"}
Sep 17 15:37:14.401757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.401467    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb" event={"ID":"2fb008fd-a61b-4e19-9c78-5dbf2acb5bec","Type":"ContainerDied","Data":"b23512b2c75994a975f47e86b51340961cc352eeceb8e9587c30664ac382b549"}
Sep 17 15:37:14.401757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.401492    2564 scope.go:117] "RemoveContainer" containerID="3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"
Sep 17 15:37:14.409911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.409895    2564 scope.go:117] "RemoveContainer" containerID="3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"
Sep 17 15:37:14.410169 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:37:14.410150    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e\": container with ID starting with 3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e not found: ID does not exist" containerID="3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"
Sep 17 15:37:14.410236 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.410181    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e"} err="failed to get container status \"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e\": rpc error: code = NotFound desc = could not find container \"3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e\": container with ID starting with 3b6c1bdb411d467bcdb4f84915a0672f2cad9558812921e5788b4bd5c8e3956e not found: ID does not exist"
Sep 17 15:37:14.454304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.454277    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:37:14.460351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.460329    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qw4gb"]
Sep 17 15:37:14.524187 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:14.524164    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" path="/var/lib/kubelet/pods/2fb008fd-a61b-4e19-9c78-5dbf2acb5bec/volumes"
Sep 17 15:37:30.246625 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.246579    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"]
Sep 17 15:37:30.247009 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.246956    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" containerName="manager"
Sep 17 15:37:30.247009 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.246969    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" containerName="manager"
Sep 17 15:37:30.247078 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.247055    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fb008fd-a61b-4e19-9c78-5dbf2acb5bec" containerName="manager"
Sep 17 15:37:30.249794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.249776    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.260820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.260802    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-b7wrn\""
Sep 17 15:37:30.274796 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.274772    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"]
Sep 17 15:37:30.363811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363780    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b7cae-1c87-492c-8404-f3495908a164-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363882    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363897    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b7cae-1c87-492c-8404-f3495908a164-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363918    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.363984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363964    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj7db\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-kube-api-access-tj7db\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.364244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.363989    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.364244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.364049    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj7db\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-kube-api-access-tj7db\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465287    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465315    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465356    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b7cae-1c87-492c-8404-f3495908a164-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465407    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b7cae-1c87-492c-8404-f3495908a164-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465589    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465788 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465877 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465848    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465939 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465890    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.465939 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465916    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.466035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.465988    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.466197 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.466177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/821b7cae-1c87-492c-8404-f3495908a164-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.467843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.467816    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/821b7cae-1c87-492c-8404-f3495908a164-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.468039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.468019    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/821b7cae-1c87-492c-8404-f3495908a164-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.490052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.490012    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.493631 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.493597    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj7db\" (UniqueName: \"kubernetes.io/projected/821b7cae-1c87-492c-8404-f3495908a164-kube-api-access-tj7db\") pod \"maas-default-gateway-openshift-default-78bb78ff78-p4swv\" (UID: \"821b7cae-1c87-492c-8404-f3495908a164\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.559962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.559928    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:30.850584 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.850439    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"]
Sep 17 15:37:30.852793 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:37:30.852771    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821b7cae_1c87_492c_8404_f3495908a164.slice/crio-af9f64c5213f789d99d5bb42dbc0529e38ff9e7078d20bfa933ce8a72f292d6e WatchSource:0}: Error finding container af9f64c5213f789d99d5bb42dbc0529e38ff9e7078d20bfa933ce8a72f292d6e: Status 404 returned error can't find the container with id af9f64c5213f789d99d5bb42dbc0529e38ff9e7078d20bfa933ce8a72f292d6e
Sep 17 15:37:30.854779 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.854749    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:37:30.854860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.854814    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:37:30.854860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:30.854843    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:37:31.482856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:31.482827    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" event={"ID":"821b7cae-1c87-492c-8404-f3495908a164","Type":"ContainerStarted","Data":"a99f15cce83dc46f0f67b91ded158ecf79a9e70617ff548f27245e6c86b2f432"}
Sep 17 15:37:31.483200 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:31.482874    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" event={"ID":"821b7cae-1c87-492c-8404-f3495908a164","Type":"ContainerStarted","Data":"af9f64c5213f789d99d5bb42dbc0529e38ff9e7078d20bfa933ce8a72f292d6e"}
Sep 17 15:37:31.510064 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:31.510005    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podStartSLOduration=1.509985275 podStartE2EDuration="1.509985275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:31.509115759 +0000 UTC m=+443.476293577" watchObservedRunningTime="2026-09-17 15:37:31.509985275 +0000 UTC m=+443.477163094"
Sep 17 15:37:31.560292 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:31.560271    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:31.565150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:31.565129    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:32.487571 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:32.487537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:32.488452 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:32.488434    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:37:33.140119 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.140082    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:37:33.142564 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.142542    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.146866 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.146843    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 15:37:33.147072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.147058    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4cmh6\""
Sep 17 15:37:33.158787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.158765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:37:33.188802 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.188772    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5llcl\" (UniqueName: \"kubernetes.io/projected/10016316-f547-4d86-a073-1fc468ec9c55-kube-api-access-5llcl\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.188927 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.188838    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10016316-f547-4d86-a073-1fc468ec9c55-config-file\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.202694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.202673    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:37:33.289500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.289480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5llcl\" (UniqueName: \"kubernetes.io/projected/10016316-f547-4d86-a073-1fc468ec9c55-kube-api-access-5llcl\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.289600 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.289531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10016316-f547-4d86-a073-1fc468ec9c55-config-file\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.290085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.290067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/10016316-f547-4d86-a073-1fc468ec9c55-config-file\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.299100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.299075    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5llcl\" (UniqueName: \"kubernetes.io/projected/10016316-f547-4d86-a073-1fc468ec9c55-kube-api-access-5llcl\") pod \"limitador-limitador-84994f7ffc-266gb\" (UID: \"10016316-f547-4d86-a073-1fc468ec9c55\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.453359 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.453292    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:33.735794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:33.735769    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:37:33.737347 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:37:33.737321    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10016316_f547_4d86_a073_1fc468ec9c55.slice/crio-02e280c250c739178e5db11a39aebb43c37154afae7bfc1b7f8a75d168e4c88b WatchSource:0}: Error finding container 02e280c250c739178e5db11a39aebb43c37154afae7bfc1b7f8a75d168e4c88b: Status 404 returned error can't find the container with id 02e280c250c739178e5db11a39aebb43c37154afae7bfc1b7f8a75d168e4c88b
Sep 17 15:37:34.499668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:34.499638    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb" event={"ID":"10016316-f547-4d86-a073-1fc468ec9c55","Type":"ContainerStarted","Data":"02e280c250c739178e5db11a39aebb43c37154afae7bfc1b7f8a75d168e4c88b"}
Sep 17 15:37:35.505559 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:35.505471    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb" event={"ID":"10016316-f547-4d86-a073-1fc468ec9c55","Type":"ContainerStarted","Data":"9dfb3ec741d0b7880ed35694ba191e9ab1bfe1d436cefab7606f2a61d3da80da"}
Sep 17 15:37:35.505928 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:35.505567    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:46.511636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:46.511584    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb"
Sep 17 15:37:46.533771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:37:46.533720    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-266gb" podStartSLOduration=12.126550856 podStartE2EDuration="13.533703637s" totalImagesPullingTime="1.407152781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:37:35.54186186 +0000 UTC m=+447.509039677" watchObservedRunningTime="2026-09-17 15:37:46.533703637 +0000 UTC m=+458.500881455"
Sep 17 15:38:51.991203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:38:51.991169    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:39:01.918958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.918922    2564 generic.go:356] "Generic (PLEG): container finished" podID="821b7cae-1c87-492c-8404-f3495908a164" containerID="a99f15cce83dc46f0f67b91ded158ecf79a9e70617ff548f27245e6c86b2f432" exitCode=0
Sep 17 15:39:01.919448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.918972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" event={"ID":"821b7cae-1c87-492c-8404-f3495908a164","Type":"ContainerDied","Data":"a99f15cce83dc46f0f67b91ded158ecf79a9e70617ff548f27245e6c86b2f432"}
Sep 17 15:39:01.919448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.919423    2564 scope.go:117] "RemoveContainer" containerID="a99f15cce83dc46f0f67b91ded158ecf79a9e70617ff548f27245e6c86b2f432"
Sep 17 15:39:01.920433 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.920406    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:39:01.920490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.920471    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:39:01.920526 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:01.920504    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:39:02.488155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:02.488119    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:39:02.929350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:02.929312    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" event={"ID":"821b7cae-1c87-492c-8404-f3495908a164","Type":"ContainerStarted","Data":"069a61b46eaf95544193c322efd2997765488780bb6c49bffcd982b810196cd5"}
Sep 17 15:39:03.560976 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:03.560949    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:39:03.562051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:03.562031    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p4swv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 17 15:39:03.562105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:03.562080    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podUID="821b7cae-1c87-492c-8404-f3495908a164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 17 15:39:04.560367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:04.560331    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p4swv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 17 15:39:04.560762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:04.560400    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podUID="821b7cae-1c87-492c-8404-f3495908a164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 17 15:39:05.560573 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:05.560535    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p4swv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 17 15:39:05.560956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:05.560603    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podUID="821b7cae-1c87-492c-8404-f3495908a164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 17 15:39:06.560599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:06.560564    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p4swv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 17 15:39:06.560997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:06.560646    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podUID="821b7cae-1c87-492c-8404-f3495908a164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 17 15:39:07.560842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:07.560803    2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-p4swv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Sep 17 15:39:07.561298 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:07.560875    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv" podUID="821b7cae-1c87-492c-8404-f3495908a164" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Sep 17 15:39:08.564073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:08.564045    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:39:08.564447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:08.564303    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:39:08.564997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:08.564983    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-p4swv"
Sep 17 15:39:14.310423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:14.310393    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:39:18.982648 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:18.982602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:39:39.264844 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.264809    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"]
Sep 17 15:39:39.267732 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.267708    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.270176 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.270145    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 15:39:39.270919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.270902    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 17 15:39:39.271023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.270967    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ndjls\""
Sep 17 15:39:39.271208 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.271195    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 15:39:39.280355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.280335    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"]
Sep 17 15:39:39.411930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.411894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.412120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.411939    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd86n\" (UniqueName: \"kubernetes.io/projected/81d320fc-7721-4649-bdb8-fa41d453aa6b-kube-api-access-nd86n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.412120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.411998    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.412120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.412017    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.412120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.412106    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d320fc-7721-4649-bdb8-fa41d453aa6b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.412318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.412177    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.512887 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.512847    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.512921    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.512953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd86n\" (UniqueName: \"kubernetes.io/projected/81d320fc-7721-4649-bdb8-fa41d453aa6b-kube-api-access-nd86n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.512990    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.513063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513303 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.513140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d320fc-7721-4649-bdb8-fa41d453aa6b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.513417    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513485 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.513443    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.513485 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.513468    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.515564 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.515505    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d320fc-7721-4649-bdb8-fa41d453aa6b-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.515564 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.515540    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d320fc-7721-4649-bdb8-fa41d453aa6b-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.523676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.523655    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd86n\" (UniqueName: \"kubernetes.io/projected/81d320fc-7721-4649-bdb8-fa41d453aa6b-kube-api-access-nd86n\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm\" (UID: \"81d320fc-7721-4649-bdb8-fa41d453aa6b\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.580861 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.580813    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:39.853326 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:39.853256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"]
Sep 17 15:39:39.855603 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:39:39.855572    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d320fc_7721_4649_bdb8_fa41d453aa6b.slice/crio-4e1b8bf3cc32e9b8b88dfc51094fb3df2d6d317bed24d46689fe9b50ec9c9034 WatchSource:0}: Error finding container 4e1b8bf3cc32e9b8b88dfc51094fb3df2d6d317bed24d46689fe9b50ec9c9034: Status 404 returned error can't find the container with id 4e1b8bf3cc32e9b8b88dfc51094fb3df2d6d317bed24d46689fe9b50ec9c9034
Sep 17 15:39:40.113194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:40.113114    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm" event={"ID":"81d320fc-7721-4649-bdb8-fa41d453aa6b","Type":"ContainerStarted","Data":"4e1b8bf3cc32e9b8b88dfc51094fb3df2d6d317bed24d46689fe9b50ec9c9034"}
Sep 17 15:39:40.468199 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:40.468124    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:39:41.121597 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:41.121499    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm" event={"ID":"81d320fc-7721-4649-bdb8-fa41d453aa6b","Type":"ContainerStarted","Data":"ee6fae86b657d8dcbdf3d95a8bd11aae437bf7b44df227e3e95618d5fae7b380"}
Sep 17 15:39:41.148663 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:41.148573    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm" podStartSLOduration=1.1886302149999999 podStartE2EDuration="2.148556844s" totalImagesPullingTime="959.926629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:41.146782531 +0000 UTC m=+573.113960349" watchObservedRunningTime="2026-09-17 15:39:41.148556844 +0000 UTC m=+573.115734663"
Sep 17 15:39:49.581839 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:49.581805    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:49.582214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:49.581890    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:49.594587 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:49.594563    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:50.174720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:50.174691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm"
Sep 17 15:39:52.010131 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:39:52.010098    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:40:03.067540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:03.067467    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:40:12.678021 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:12.677984    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:40:24.668308 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.668276    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"]
Sep 17 15:40:24.671078 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.671063    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.673312 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.673291    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 15:40:24.683174 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.683152    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"]
Sep 17 15:40:24.819824 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.819784    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jrrq\" (UniqueName: \"kubernetes.io/projected/46ec7d99-22be-4596-89c2-f9eaa953febb-kube-api-access-8jrrq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.819824 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.819830    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.819997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.819899    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.819997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.819933    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.819997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.819985    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.820093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.820057    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ec7d99-22be-4596-89c2-f9eaa953febb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920517 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ec7d99-22be-4596-89c2-f9eaa953febb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920517 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920477    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jrrq\" (UniqueName: \"kubernetes.io/projected/46ec7d99-22be-4596-89c2-f9eaa953febb-kube-api-access-8jrrq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920517 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920500    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920678    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920717    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920797    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.920953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.920906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.921080 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.921055    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.921194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.921137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.922798 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.922776    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46ec7d99-22be-4596-89c2-f9eaa953febb-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.922957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.922940    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46ec7d99-22be-4596-89c2-f9eaa953febb-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.930159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.930140    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jrrq\" (UniqueName: \"kubernetes.io/projected/46ec7d99-22be-4596-89c2-f9eaa953febb-kube-api-access-8jrrq\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-22g7j\" (UID: \"46ec7d99-22be-4596-89c2-f9eaa953febb\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:24.980082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:24.980062    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:25.250439 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:25.250369    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"]
Sep 17 15:40:25.253530 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:40:25.253501    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46ec7d99_22be_4596_89c2_f9eaa953febb.slice/crio-84d4bc79a187061461bc874073c082ef297ac88717bb0e7376f310eb9785a142 WatchSource:0}: Error finding container 84d4bc79a187061461bc874073c082ef297ac88717bb0e7376f310eb9785a142: Status 404 returned error can't find the container with id 84d4bc79a187061461bc874073c082ef297ac88717bb0e7376f310eb9785a142
Sep 17 15:40:25.254950 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:25.254933    2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:40:25.346032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:25.346002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j" event={"ID":"46ec7d99-22be-4596-89c2-f9eaa953febb","Type":"ContainerStarted","Data":"84d4bc79a187061461bc874073c082ef297ac88717bb0e7376f310eb9785a142"}
Sep 17 15:40:26.352291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:26.352260    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j" event={"ID":"46ec7d99-22be-4596-89c2-f9eaa953febb","Type":"ContainerStarted","Data":"8e493ab82d769ed65ea4bf703600b777c87721729f57678b87b788f4f00093d5"}
Sep 17 15:40:26.373593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:26.373542    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j" podStartSLOduration=2.205082848 podStartE2EDuration="2.373527231s" totalImagesPullingTime="168.444383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:26.371425228 +0000 UTC m=+618.338603046" watchObservedRunningTime="2026-09-17 15:40:26.373527231 +0000 UTC m=+618.340705086"
Sep 17 15:40:27.030968 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:27.030939    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:40:34.980370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:34.980335    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:34.980770 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:34.980384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:34.992570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:34.992548    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:40:35.414302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:40:35.414273    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-22g7j"
Sep 17 15:42:21.697388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.697352    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:42:21.701071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.701044    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.703814 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.703793    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c86e9a-openshift-default-dockercfg-6zrsd\""
Sep 17 15:42:21.722458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.722431    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:42:21.806861 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.806830    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:42:21.810975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.810956    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.813258 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.813239    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-449c47-openshift-default-dockercfg-9vds5\""
Sep 17 15:42:21.825080 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.825060    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:42:21.853031 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853007    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853123 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853123 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853080    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853199 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853228    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htww\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853260    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853359 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853345    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853398 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853373    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.853442 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.853399    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.954659 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954536    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htww\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.954659 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954595    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954753    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954805    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954839    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954910    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.954979    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955005    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pd2\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955034    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955069    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955114    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955144    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955192    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955224    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955265    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:21.955958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955301    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.955958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955774    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.957730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.956060    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.957730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.956396    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.957730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.956556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.957730 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.955786    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.959898 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.959589    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.959898 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.959835    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.974011 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.973954    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:21.974604 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:21.974576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htww\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww\") pod \"e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") " pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:22.012420 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.012388    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:22.056072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056034    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056091    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056145    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056199    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056237    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056275    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pd2\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056325    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.056383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056353    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.057151 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.056898    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.057151 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.057119    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.057151 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.057136    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.057326 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.057178    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.057953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.057930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.061956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.061936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.063855 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.063833    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.071059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.070959    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.074629 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.074596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pd2\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2\") pod \"e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") " pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.121130 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.121094    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:22.299967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.299941    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:42:22.301506 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:22.301480    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11685c29_4640_442d_a39c_94291e185ea7.slice/crio-1a4e684b81e37d11ab398082fcee1906edc3c9c178deabe52b797213c1fb446a WatchSource:0}: Error finding container 1a4e684b81e37d11ab398082fcee1906edc3c9c178deabe52b797213c1fb446a: Status 404 returned error can't find the container with id 1a4e684b81e37d11ab398082fcee1906edc3c9c178deabe52b797213c1fb446a
Sep 17 15:42:22.303659 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.303629    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.303734 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.303690    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.303734 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.303718    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.402337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.402310    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:42:22.405371 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:22.405343    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod735434a5_d970_4b74_88ec_c6cd5148a1d0.slice/crio-a2872c613c3a99bace14049fecf1d6fa86222aa9472fb008e7397993d665a205 WatchSource:0}: Error finding container a2872c613c3a99bace14049fecf1d6fa86222aa9472fb008e7397993d665a205: Status 404 returned error can't find the container with id a2872c613c3a99bace14049fecf1d6fa86222aa9472fb008e7397993d665a205
Sep 17 15:42:22.408087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.408045    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.408172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.408154    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.408214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.408201    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:22.913842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.913806    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerStarted","Data":"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"}
Sep 17 15:42:22.914277 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.913869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerStarted","Data":"a2872c613c3a99bace14049fecf1d6fa86222aa9472fb008e7397993d665a205"}
Sep 17 15:42:22.915367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.915344    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerStarted","Data":"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"}
Sep 17 15:42:22.915438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.915392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerStarted","Data":"1a4e684b81e37d11ab398082fcee1906edc3c9c178deabe52b797213c1fb446a"}
Sep 17 15:42:22.939652 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.939580    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podStartSLOduration=1.9395666679999999 podStartE2EDuration="1.939566668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:22.937529024 +0000 UTC m=+734.904706835" watchObservedRunningTime="2026-09-17 15:42:22.939566668 +0000 UTC m=+734.906744486"
Sep 17 15:42:22.973846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:22.973801    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podStartSLOduration=1.97378388 podStartE2EDuration="1.97378388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:22.969301573 +0000 UTC m=+734.936479392" watchObservedRunningTime="2026-09-17 15:42:22.97378388 +0000 UTC m=+734.940961702"
Sep 17 15:42:23.012755 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.012726    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:23.017824 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.017804    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:23.121676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.121636    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:23.123155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.123132    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:23.123245 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.123181    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:23.920663 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.920602    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:23.921838 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:23.921817    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:24.125490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:24.125462    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:24.926288 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:24.926253    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:24.927490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:24.927469    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:25.202905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:25.202815    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:42:29.763179 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.763148    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:42:29.767980 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.767958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.770291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.770268    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-28eba6-openshift-default-dockercfg-zx89s\""
Sep 17 15:42:29.785501 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.785480    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:42:29.937596 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937562    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937665    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937698    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph96w\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937783    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937816    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:29.937971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:29.937887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.039218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.039103    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.039489 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.039465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.039767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.039749    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.039952 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.039935    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph96w\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040260 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040244    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040461 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.039768    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040588 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040571    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040713    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.040893 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040876    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.041084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.040596    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.041161 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.041129    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.041656 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.041605    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.041766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.041674    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.043498 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.043476    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.044018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.043997    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.054944 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.054923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph96w\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.056106 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.056086    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token\") pod \"e2e-autores-28eba6-openshift-default-544d775478-s6m9w\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") " pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.078873 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.078846    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:30.358350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.358328    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:42:30.360444 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:30.360416    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aaa06c9_115f_4c2a_9f96_3dc7c95842f7.slice/crio-11aef9cf9dcd661e8ff996e813f5763ca61900e13705668b7885d7d0c0211c12 WatchSource:0}: Error finding container 11aef9cf9dcd661e8ff996e813f5763ca61900e13705668b7885d7d0c0211c12: Status 404 returned error can't find the container with id 11aef9cf9dcd661e8ff996e813f5763ca61900e13705668b7885d7d0c0211c12
Sep 17 15:42:30.362419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.362392    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:30.362490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.362447    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:30.362490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.362477    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:30.960683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.960646    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerStarted","Data":"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"}
Sep 17 15:42:30.961133 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.960698    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerStarted","Data":"11aef9cf9dcd661e8ff996e813f5763ca61900e13705668b7885d7d0c0211c12"}
Sep 17 15:42:30.986057 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:30.986003    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podStartSLOduration=1.985988502 podStartE2EDuration="1.985988502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:30.98460819 +0000 UTC m=+742.951786007" watchObservedRunningTime="2026-09-17 15:42:30.985988502 +0000 UTC m=+742.953166320"
Sep 17 15:42:31.079405 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.079382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:31.084606 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.084585    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:31.966057 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.966026    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:31.967014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.966997    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:42:31.994220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.994193    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:42:31.998007 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:31.997987    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.001907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.001886    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-76a88377-openshift-default-dockercfg-nnrlk\""
Sep 17 15:42:32.020668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.020637    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:42:32.161534 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.161767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161560    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.161767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161591    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.161767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161666    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.161767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161719    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.162010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161796    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.162010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161859    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mksz9\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.162010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161905    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.162010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.161931    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.262760 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262727    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.262931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262777    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.262931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262800    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.262931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262844    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.262931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262889    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.262954    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263014    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mksz9\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263089    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263353 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263353 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263483 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263453    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263596 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263576    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.263727 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.263711    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.265290 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.265271    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.265588 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.265569    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.273605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.273582    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.273853 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.273836    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mksz9\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9\") pod \"e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") " pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.312424 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.312402    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:32.593927 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.593903    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:42:32.595655 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:32.595622    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f260867_9930_491d_9c85_8aca6ed5a5c5.slice/crio-b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72 WatchSource:0}: Error finding container b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72: Status 404 returned error can't find the container with id b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72
Sep 17 15:42:32.598146 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.598107    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:32.598221 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.598201    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:32.598276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.598247    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:32.972947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.972867    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerStarted","Data":"5c5d8240afa1619e8ded04de78759b2a6b64742bebd850030feab77e5b2471a8"}
Sep 17 15:42:32.972947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:32.972929    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerStarted","Data":"b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72"}
Sep 17 15:42:33.001516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:33.001465    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podStartSLOduration=2.001450343 podStartE2EDuration="2.001450343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:32.998715202 +0000 UTC m=+744.965893043" watchObservedRunningTime="2026-09-17 15:42:33.001450343 +0000 UTC m=+744.968628160"
Sep 17 15:42:33.312842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:33.312813    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:33.319986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:33.319957    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:33.978042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:33.978014    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:33.978886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:33.978868    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:42:39.962793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:39.962747    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 15:42:39.963186 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:39.962814    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 15:42:40.001626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.001569    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:42:40.537238 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.537210    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:42:40.540465 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.540447    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.546579 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.546553    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a2dc23-openshift-default-dockercfg-4mpcj\""
Sep 17 15:42:40.577049 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.576880    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577049 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.576935    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577049 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.576967    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577049 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.576999    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.577147    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.577183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.577232    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.577282    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj56n\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.577572 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.577439    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.584605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.584566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:42:40.679002 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.678968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679007    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679053    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679313 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679192    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj56n\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679313 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679267    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679354    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679395    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679430    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679462    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679525    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679772    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679874 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679461    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.679935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679882    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.680007 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.679989    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.681949 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.681921    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.682057 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.682039    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.693399 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.693372    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.694664 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.694638    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj56n\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n\") pod \"e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") " pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.852601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.852513    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:40.968387 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.968275    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 15:42:40.968870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:40.968402    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 15:42:41.160278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.160187    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:42:41.166072 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:41.166038    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1fbbd0_b275_4843_ae48_8978d1e2fca1.slice/crio-1953e75ce757f60184497447f88733eb21f1d4bdec64e6d5aadb8c48a8b79bf9 WatchSource:0}: Error finding container 1953e75ce757f60184497447f88733eb21f1d4bdec64e6d5aadb8c48a8b79bf9: Status 404 returned error can't find the container with id 1953e75ce757f60184497447f88733eb21f1d4bdec64e6d5aadb8c48a8b79bf9
Sep 17 15:42:41.168672 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.168595    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:41.168776 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.168725    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:41.168841 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.168773    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:41.578538 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.578497    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:42:41.582792 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.582768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.585228 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.585198    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-76a88377-dockercfg-mg7gr\""
Sep 17 15:42:41.585370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.585241    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-76a88377\""
Sep 17 15:42:41.601353 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.601322    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:42:41.699511 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.699469    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.699867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.699842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzp6d\" (UniqueName: \"kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.800868 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.800836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.801034 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.800896    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzp6d\" (UniqueName: \"kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.801609 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.801587    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.816203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.816123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzp6d\" (UniqueName: \"kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d\") pod \"payload-processing-e2e-ait-76a88377-86847f66c7-v97bw\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") " pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:41.916287 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:41.916182    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:42.059476 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.059206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerStarted","Data":"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"}
Sep 17 15:42:42.059476 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.059280    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerStarted","Data":"1953e75ce757f60184497447f88733eb21f1d4bdec64e6d5aadb8c48a8b79bf9"}
Sep 17 15:42:42.087052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.086987    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podStartSLOduration=2.086960752 podStartE2EDuration="2.086960752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:42.082388283 +0000 UTC m=+754.049566104" watchObservedRunningTime="2026-09-17 15:42:42.086960752 +0000 UTC m=+754.054138572"
Sep 17 15:42:42.242525 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.242496    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:42:42.244927 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:42:42.244900    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021802a5_1d2e_45d2_9e8e_29142f98bed3.slice/crio-994ef8fe00da914a8e3705cf13b019873e62611569e71c32646258bace9b22b3 WatchSource:0}: Error finding container 994ef8fe00da914a8e3705cf13b019873e62611569e71c32646258bace9b22b3: Status 404 returned error can't find the container with id 994ef8fe00da914a8e3705cf13b019873e62611569e71c32646258bace9b22b3
Sep 17 15:42:42.853707 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.853671    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:42.859688 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:42.859657    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:43.074012 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.073984    2564 generic.go:356] "Generic (PLEG): container finished" podID="11685c29-4640-442d-a39c-94291e185ea7" containerID="6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108" exitCode=0
Sep 17 15:42:43.074428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.074068    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerDied","Data":"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"}
Sep 17 15:42:43.074723 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.074687    2564 scope.go:117] "RemoveContainer" containerID="6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"
Sep 17 15:42:43.077096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.076244    2564 generic.go:356] "Generic (PLEG): container finished" podID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerID="86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94" exitCode=0
Sep 17 15:42:43.077096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.076396    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerDied","Data":"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"}
Sep 17 15:42:43.077096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.076887    2564 scope.go:117] "RemoveContainer" containerID="86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"
Sep 17 15:42:43.078185 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.078164    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" event={"ID":"021802a5-1d2e-45d2-9e8e-29142f98bed3","Type":"ContainerStarted","Data":"994ef8fe00da914a8e3705cf13b019873e62611569e71c32646258bace9b22b3"}
Sep 17 15:42:43.078496 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.078479    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:43.079453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.079433    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:42:43.378667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378564    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:43.378667 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378664    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:43.378843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378672    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:43.378843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378696    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:43.378843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378739    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:43.378843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:43.378770    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:42:44.087771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.087732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" event={"ID":"021802a5-1d2e-45d2-9e8e-29142f98bed3","Type":"ContainerStarted","Data":"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"}
Sep 17 15:42:44.088248 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.087802    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:42:44.090154 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.090128    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerStarted","Data":"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"}
Sep 17 15:42:44.092372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.092345    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerStarted","Data":"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"}
Sep 17 15:42:44.121545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.121522    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:44.121750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.121724    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:44.121861 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.121772    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:44.146865 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:44.146787    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" podStartSLOduration=1.9726786349999998 podStartE2EDuration="3.146768128s" totalImagesPullingTime="1.174089493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:44.115358701 +0000 UTC m=+756.082536521" watchObservedRunningTime="2026-09-17 15:42:44.146768128 +0000 UTC m=+756.113945948"
Sep 17 15:42:45.013799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:45.013756    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:45.014011 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:45.013912    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 15:42:45.014011 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:45.013961    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 15:42:45.122078 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:45.122040    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:45.122696 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:45.122663    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:46.013382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:46.013335    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 15:42:46.013591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:46.013404    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 15:42:46.121570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:46.121536    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:46.121852 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:46.121823    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:47.013259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:47.012811    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 15:42:47.013259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:47.012876    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 15:42:47.123364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:47.122958    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:47.123364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:47.123014    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:48.013872 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:48.013824    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 15:42:48.014365 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:48.013899    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 15:42:48.122242 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:48.122206    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:48.122514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:48.122484    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:49.013023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:49.012988    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body=
Sep 17 15:42:49.013291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:49.013262    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused"
Sep 17 15:42:49.121749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:49.121714    2564 patch_prober.go:28] interesting pod/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body=
Sep 17 15:42:49.122357 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:49.122316    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused"
Sep 17 15:42:50.075686 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.075648    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:50.076179 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.076153    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:50.077804 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.077780    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:42:50.125785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.125758    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:50.126208 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.126047    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:50.126602 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:50.126583    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:42:55.099802 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:42:55.099769    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:43:00.224774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.224720    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"]
Sep 17 15:43:00.229123 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.229099    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:00.231416 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.231392    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:43:00.231786 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.231767    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:43:00.236019 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.235995    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"]
Sep 17 15:43:00.325398 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.325356    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a88377-qpcpr\" (UID: \"94fe1a15-8e04-4e13-82b0-a1261cbe1074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:00.426743 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.426713    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a88377-qpcpr\" (UID: \"94fe1a15-8e04-4e13-82b0-a1261cbe1074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:00.427273 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.427253    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-76a88377-qpcpr\" (UID: \"94fe1a15-8e04-4e13-82b0-a1261cbe1074\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:00.541834 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.541809    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:00.831483 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:00.831451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"]
Sep 17 15:43:00.833885 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:43:00.833847    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fe1a15_8e04_4e13_82b0_a1261cbe1074.slice/crio-65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce WatchSource:0}: Error finding container 65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce: Status 404 returned error can't find the container with id 65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce
Sep 17 15:43:01.209145 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.209066    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerID="46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c" exitCode=0
Sep 17 15:43:01.209303 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.209156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerDied","Data":"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"}
Sep 17 15:43:01.209794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.209766    2564 scope.go:117] "RemoveContainer" containerID="46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"
Sep 17 15:43:01.210638 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.210592    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr" event={"ID":"94fe1a15-8e04-4e13-82b0-a1261cbe1074","Type":"ContainerStarted","Data":"65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce"}
Sep 17 15:43:01.211304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.211278    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:01.211357 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.211346    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:01.211395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.211378    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:01.966208 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:01.966139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:43:02.220526 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.220440    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerStarted","Data":"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"}
Sep 17 15:43:02.222486 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.222454    2564 generic.go:356] "Generic (PLEG): container finished" podID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerID="5c5d8240afa1619e8ded04de78759b2a6b64742bebd850030feab77e5b2471a8" exitCode=0
Sep 17 15:43:02.222634 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.222501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerDied","Data":"5c5d8240afa1619e8ded04de78759b2a6b64742bebd850030feab77e5b2471a8"}
Sep 17 15:43:02.222969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.222950    2564 scope.go:117] "RemoveContainer" containerID="5c5d8240afa1619e8ded04de78759b2a6b64742bebd850030feab77e5b2471a8"
Sep 17 15:43:02.224235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.224204    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:02.224344 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.224289    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:02.224409 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:02.224340    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:03.079815 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.079778    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:43:03.081129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.081101    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:03.081249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.081154    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:03.256638 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.255680    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerStarted","Data":"e581a705a37841f351e606e43ef5ff1ab7ffee1b699ca5bf71019f9db21f1535"}
Sep 17 15:43:03.315863 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.315820    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:43:03.318540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.318510    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:03.318807 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.318725    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:03.318807 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.318784    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:03.978709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:03.978538    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:04.079536 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.079495    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:04.079762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.079560    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:04.267754 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.267702    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr" event={"ID":"94fe1a15-8e04-4e13-82b0-a1261cbe1074","Type":"ContainerStarted","Data":"894798d62a5b3eb8e894ffac620a5b371d9dc1a80dc7bb2bc8d39ba41d52e469"}
Sep 17 15:43:04.289989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.289912    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr" podStartSLOduration=1.397095057 podStartE2EDuration="4.289888958s" totalImagesPullingTime="2.892793901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:04.288196099 +0000 UTC m=+776.255373917" watchObservedRunningTime="2026-09-17 15:43:04.289888958 +0000 UTC m=+776.257066774"
Sep 17 15:43:04.313193 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.313154    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:04.313377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:04.313221    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:05.079244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.079206    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:05.079458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.079271    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:05.292724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.292682    2564 generic.go:356] "Generic (PLEG): container finished" podID="94fe1a15-8e04-4e13-82b0-a1261cbe1074" containerID="894798d62a5b3eb8e894ffac620a5b371d9dc1a80dc7bb2bc8d39ba41d52e469" exitCode=0
Sep 17 15:43:05.293220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.292720    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr" event={"ID":"94fe1a15-8e04-4e13-82b0-a1261cbe1074","Type":"ContainerDied","Data":"894798d62a5b3eb8e894ffac620a5b371d9dc1a80dc7bb2bc8d39ba41d52e469"}
Sep 17 15:43:05.313056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.313022    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:05.313331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:05.313300    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:06.079929 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.079884    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:06.080140 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.080000    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:06.314636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.313231    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:06.314636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.313288    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:06.638740 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.638431    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:06.801967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.801932    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca\") pod \"94fe1a15-8e04-4e13-82b0-a1261cbe1074\" (UID: \"94fe1a15-8e04-4e13-82b0-a1261cbe1074\") "
Sep 17 15:43:06.803062 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.803033    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94fe1a15-8e04-4e13-82b0-a1261cbe1074" (UID: "94fe1a15-8e04-4e13-82b0-a1261cbe1074"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:06.903707 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:06.903676    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fe1a15-8e04-4e13-82b0-a1261cbe1074-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:07.079568 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.079470    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:07.079870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.079832    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:07.305687 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.305649    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr" event={"ID":"94fe1a15-8e04-4e13-82b0-a1261cbe1074","Type":"ContainerDied","Data":"65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce"}
Sep 17 15:43:07.306136 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.306108    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65820cfc31e8844fe9def4c07d85635ef99f39f645db8716981b0a1658d219ce"
Sep 17 15:43:07.306908 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.305962    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"
Sep 17 15:43:07.313310 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.312908    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:07.313310 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:07.312961    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:08.079741 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:08.079696    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Sep 17 15:43:08.080218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:08.079767    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Sep 17 15:43:08.314756 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:08.314715    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Sep 17 15:43:08.315034 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:08.315008    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Sep 17 15:43:09.084081 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:09.084050    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:43:09.084468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:09.084419    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:43:09.085155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:09.085139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:43:09.325261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:09.325229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:09.330326 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:09.330299    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:13.538057 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:13.538025    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:54846->10.133.0.38:15021: read: connection reset by peer" start-of-body=
Sep 17 15:43:13.538485 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:13.538092    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:54846->10.133.0.38:15021: read: connection reset by peer"
Sep 17 15:43:14.351777 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.351745    2564 generic.go:356] "Generic (PLEG): container finished" podID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerID="3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b" exitCode=0
Sep 17 15:43:14.351968 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.351822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerDied","Data":"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"}
Sep 17 15:43:14.352281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.352265    2564 scope.go:117] "RemoveContainer" containerID="3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"
Sep 17 15:43:14.353376 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.353352    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:14.353428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.353417    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:14.353463 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:14.353451    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:15.363402 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:15.363362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerStarted","Data":"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"}
Sep 17 15:43:15.853116 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:15.853084    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:43:15.853321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:15.853298    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 15:43:15.853384 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:15.853366    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 15:43:16.853101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:16.853067    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 15:43:16.853465 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:16.853149    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 15:43:17.853272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:17.853234    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 15:43:17.853641 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:17.853293    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 15:43:18.853606 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:18.853570    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 15:43:18.853981 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:18.853670    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 15:43:19.853159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:19.853123    2564 patch_prober.go:28] interesting pod/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 17 15:43:19.853336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:19.853198    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 17 15:43:20.857298 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:20.857267    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:43:20.857782 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:20.857643    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:43:20.858413 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:20.858395    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:43:36.808379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:36.808347    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:43:36.809274 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:36.808596    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" podUID="021802a5-1d2e-45d2-9e8e-29142f98bed3" containerName="payload-processing" containerID="cri-o://90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d" gracePeriod=30
Sep 17 15:43:37.028786 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.028755    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"]
Sep 17 15:43:37.032662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.032638    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-76a88377-qpcpr"]
Sep 17 15:43:37.093456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.093188    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:43:37.173783 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.173750    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:43:37.174279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.174242    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" containerID="cri-o://e581a705a37841f351e606e43ef5ff1ab7ffee1b699ca5bf71019f9db21f1535" gracePeriod=30
Sep 17 15:43:37.231986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.231946    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzp6d\" (UniqueName: \"kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d\") pod \"021802a5-1d2e-45d2-9e8e-29142f98bed3\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") "
Sep 17 15:43:37.232181 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.232007    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume\") pod \"021802a5-1d2e-45d2-9e8e-29142f98bed3\" (UID: \"021802a5-1d2e-45d2-9e8e-29142f98bed3\") "
Sep 17 15:43:37.232732 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.232689    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "021802a5-1d2e-45d2-9e8e-29142f98bed3" (UID: "021802a5-1d2e-45d2-9e8e-29142f98bed3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:37.234456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.234427    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d" (OuterVolumeSpecName: "kube-api-access-qzp6d") pod "021802a5-1d2e-45d2-9e8e-29142f98bed3" (UID: "021802a5-1d2e-45d2-9e8e-29142f98bed3"). InnerVolumeSpecName "kube-api-access-qzp6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:37.333217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.333123    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzp6d\" (UniqueName: \"kubernetes.io/projected/021802a5-1d2e-45d2-9e8e-29142f98bed3-kube-api-access-qzp6d\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.333217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.333166    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/021802a5-1d2e-45d2-9e8e-29142f98bed3-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:37.482733 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.482698    2564 generic.go:356] "Generic (PLEG): container finished" podID="021802a5-1d2e-45d2-9e8e-29142f98bed3" containerID="90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d" exitCode=0
Sep 17 15:43:37.482891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.482749    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"
Sep 17 15:43:37.482891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.482768    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" event={"ID":"021802a5-1d2e-45d2-9e8e-29142f98bed3","Type":"ContainerDied","Data":"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"}
Sep 17 15:43:37.482891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.482820    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw" event={"ID":"021802a5-1d2e-45d2-9e8e-29142f98bed3","Type":"ContainerDied","Data":"994ef8fe00da914a8e3705cf13b019873e62611569e71c32646258bace9b22b3"}
Sep 17 15:43:37.482891 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.482849    2564 scope.go:117] "RemoveContainer" containerID="90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"
Sep 17 15:43:37.491738 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.491722    2564 scope.go:117] "RemoveContainer" containerID="90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"
Sep 17 15:43:37.492000 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:43:37.491980    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d\": container with ID starting with 90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d not found: ID does not exist" containerID="90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"
Sep 17 15:43:37.492052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.492008    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d"} err="failed to get container status \"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d\": rpc error: code = NotFound desc = could not find container \"90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d\": container with ID starting with 90f50a064e8c3988b9f0cd8ee468d91e7956ff9023d58d46f6490ba5b2a8b08d not found: ID does not exist"
Sep 17 15:43:37.508297 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.508275    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:43:37.512385 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:37.512366    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-76a88377-86847f66c7-v97bw"]
Sep 17 15:43:38.242984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.242958    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:43:38.243378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243365    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="021802a5-1d2e-45d2-9e8e-29142f98bed3" containerName="payload-processing"
Sep 17 15:43:38.243378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243378    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="021802a5-1d2e-45d2-9e8e-29142f98bed3" containerName="payload-processing"
Sep 17 15:43:38.243474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243392    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94fe1a15-8e04-4e13-82b0-a1261cbe1074" containerName="revoke-keys"
Sep 17 15:43:38.243474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243398    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="94fe1a15-8e04-4e13-82b0-a1261cbe1074" containerName="revoke-keys"
Sep 17 15:43:38.243474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243456    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="021802a5-1d2e-45d2-9e8e-29142f98bed3" containerName="payload-processing"
Sep 17 15:43:38.243474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.243464    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="94fe1a15-8e04-4e13-82b0-a1261cbe1074" containerName="revoke-keys"
Sep 17 15:43:38.246035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.246019    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.248946 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.248925    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-49b90ab7-gw-openshift-default-dockercfg-fpjtm\""
Sep 17 15:43:38.262030 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.262008    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:43:38.342632 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342718    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342807 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342786    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342821    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb7xl\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342881 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342846    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342881 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342873    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.342965 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342889    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.343010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342962    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.343010 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.342983    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.444719 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444675    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.444900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444771    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.444900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444810    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb7xl\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.444900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444848    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.444900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444898    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.445125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444923    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.445125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.444975    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.445125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.445008    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.445125 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.445037    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.446521 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.446488    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.447290 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.447259    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.448643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.447825    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.448643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.447956    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.448643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.448506    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.449370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.449322    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.450790 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.450765    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.458725 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.458701    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb7xl\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.459246 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.459224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token\") pod \"e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") " pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.532728 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.532692    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="021802a5-1d2e-45d2-9e8e-29142f98bed3" path="/var/lib/kubelet/pods/021802a5-1d2e-45d2-9e8e-29142f98bed3/volumes"
Sep 17 15:43:38.533102 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.533088    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94fe1a15-8e04-4e13-82b0-a1261cbe1074" path="/var/lib/kubelet/pods/94fe1a15-8e04-4e13-82b0-a1261cbe1074/volumes"
Sep 17 15:43:38.556208 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.556184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:38.836015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.835985    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:43:38.837860 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:43:38.837820    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1375ddd9_ab6d_42f1_9a89_c6502dd93d98.slice/crio-d69570919d6bf0f0e3976215cbd2d4c1a2559c2e1f69fb67e64ad5c304520cbe WatchSource:0}: Error finding container d69570919d6bf0f0e3976215cbd2d4c1a2559c2e1f69fb67e64ad5c304520cbe: Status 404 returned error can't find the container with id d69570919d6bf0f0e3976215cbd2d4c1a2559c2e1f69fb67e64ad5c304520cbe
Sep 17 15:43:38.840904 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.840867    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:38.840986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.840946    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:38.841035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:38.840988    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:43:39.328683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.328652    2564 patch_prober.go:28] interesting pod/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:43:39.329042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.328700    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:43:39.495940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.495906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerStarted","Data":"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"}
Sep 17 15:43:39.496093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.495957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerStarted","Data":"d69570919d6bf0f0e3976215cbd2d4c1a2559c2e1f69fb67e64ad5c304520cbe"}
Sep 17 15:43:39.516913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.516865    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podStartSLOduration=1.516848645 podStartE2EDuration="1.516848645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:39.515340472 +0000 UTC m=+811.482518291" watchObservedRunningTime="2026-09-17 15:43:39.516848645 +0000 UTC m=+811.484026464"
Sep 17 15:43:39.556883 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.556849    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:39.562016 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:39.561996    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:40.501016 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:40.500986    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:40.501909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:40.501888    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:43:41.884442 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:41.884412    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:43:42.515673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.515609    2564 generic.go:356] "Generic (PLEG): container finished" podID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerID="e581a705a37841f351e606e43ef5ff1ab7ffee1b699ca5bf71019f9db21f1535" exitCode=0
Sep 17 15:43:42.515829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.515662    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerDied","Data":"e581a705a37841f351e606e43ef5ff1ab7ffee1b699ca5bf71019f9db21f1535"}
Sep 17 15:43:42.515829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.515740    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx" event={"ID":"9f260867-9930-491d-9c85-8aca6ed5a5c5","Type":"ContainerDied","Data":"b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72"}
Sep 17 15:43:42.515829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.515769    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4810e94fd12eee187a6b660a432b23bed5538e0e9f2bd5c518409be242bff72"
Sep 17 15:43:42.515829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.515785    2564 scope.go:117] "RemoveContainer" containerID="5c5d8240afa1619e8ded04de78759b2a6b64742bebd850030feab77e5b2471a8"
Sep 17 15:43:42.532850 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.532834    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:42.690991 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.690911    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.690991 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.690961    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691039    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691060    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691112    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mksz9\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691185    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691243    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy\") pod \"9f260867-9930-491d-9c85-8aca6ed5a5c5\" (UID: \"9f260867-9930-491d-9c85-8aca6ed5a5c5\") "
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691326    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691391    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691426    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691423    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691589    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691686    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691705    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691717    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f260867-9930-491d-9c85-8aca6ed5a5c5-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.691811 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.691725    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.693330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.693310    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:43:42.693427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.693347    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:43:42.693604 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.693585    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9" (OuterVolumeSpecName: "kube-api-access-mksz9") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "kube-api-access-mksz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:42.693724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.693609    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f260867-9930-491d-9c85-8aca6ed5a5c5" (UID: "9f260867-9930-491d-9c85-8aca6ed5a5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:43:42.792443 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.792416    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.792443 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.792440    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.792605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.792451    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.792605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.792459    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mksz9\" (UniqueName: \"kubernetes.io/projected/9f260867-9930-491d-9c85-8aca6ed5a5c5-kube-api-access-mksz9\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:42.792605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:42.792468    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f260867-9930-491d-9c85-8aca6ed5a5c5-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:43:43.195643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.195583    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:43:43.196076 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196061    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.196120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196077    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.196120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196087    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.196120 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196092    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.196229 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196169    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.196229 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.196178    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" containerName="istio-proxy"
Sep 17 15:43:43.200555 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.200536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.202979 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.202957    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-49b90ab7\""
Sep 17 15:43:43.203092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.203068    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-49b90ab7-dockercfg-km6rc\""
Sep 17 15:43:43.214580 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.213256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:43:43.248350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.248318    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:43:43.251871 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.251854    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.264500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.264475    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:43:43.299293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.299245    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvw65\" (UniqueName: \"kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.299484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.299359    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.400773 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.400722    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zhg\" (UniqueName: \"kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.400988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.400817    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvw65\" (UniqueName: \"kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.400988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.400842    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.401121 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.401000    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.401709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.401688    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.410175 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.410133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvw65\" (UniqueName: \"kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65\") pod \"payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") " pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.502386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.502276    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.502568 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.502460    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zhg\" (UniqueName: \"kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.503357 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.503330    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.514053 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.514016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:43.522343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.522313    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zhg\" (UniqueName: \"kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg\") pod \"payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.529959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.529893    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"
Sep 17 15:43:43.561918 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.561598    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:43.576123 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.576087    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:43:43.584542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.584515    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-76a88377-openshift-default-67587b76f4-gzgtx"]
Sep 17 15:43:43.811544 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:43:43.811518    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb926977d_d35d_4fcb_9d0d_1700e558f91b.slice/crio-bd7a794c38196dfd0f8a80eb51df7384658357be741da8e4d3b3252cc08f1e37 WatchSource:0}: Error finding container bd7a794c38196dfd0f8a80eb51df7384658357be741da8e4d3b3252cc08f1e37: Status 404 returned error can't find the container with id bd7a794c38196dfd0f8a80eb51df7384658357be741da8e4d3b3252cc08f1e37
Sep 17 15:43:43.811781 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.811755    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:43:43.857868 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:43.857845    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:43:43.859350 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:43:43.859329    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36d8e62f_b95b_489f_bc35_3180107c6fa9.slice/crio-e19ab1d3158137ddd51935a847ee4a9e93beca7b6519ad8c9bf087654e86bdf5 WatchSource:0}: Error finding container e19ab1d3158137ddd51935a847ee4a9e93beca7b6519ad8c9bf087654e86bdf5: Status 404 returned error can't find the container with id e19ab1d3158137ddd51935a847ee4a9e93beca7b6519ad8c9bf087654e86bdf5
Sep 17 15:43:44.529986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.529949    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f260867-9930-491d-9c85-8aca6ed5a5c5" path="/var/lib/kubelet/pods/9f260867-9930-491d-9c85-8aca6ed5a5c5/volumes"
Sep 17 15:43:44.539269 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.539233    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" event={"ID":"36d8e62f-b95b-489f-bc35-3180107c6fa9","Type":"ContainerStarted","Data":"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"}
Sep 17 15:43:44.539434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.539296    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" event={"ID":"36d8e62f-b95b-489f-bc35-3180107c6fa9","Type":"ContainerStarted","Data":"e19ab1d3158137ddd51935a847ee4a9e93beca7b6519ad8c9bf087654e86bdf5"}
Sep 17 15:43:44.539434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.539325    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:44.541055 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.541028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" event={"ID":"b926977d-d35d-4fcb-9d0d-1700e558f91b","Type":"ContainerStarted","Data":"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"}
Sep 17 15:43:44.541159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.541081    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" event={"ID":"b926977d-d35d-4fcb-9d0d-1700e558f91b","Type":"ContainerStarted","Data":"bd7a794c38196dfd0f8a80eb51df7384658357be741da8e4d3b3252cc08f1e37"}
Sep 17 15:43:44.541159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.541113    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:43:44.561928 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.561879    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" podStartSLOduration=1.5618660690000001 podStartE2EDuration="1.561866069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:44.559871626 +0000 UTC m=+816.527049444" watchObservedRunningTime="2026-09-17 15:43:44.561866069 +0000 UTC m=+816.529043887"
Sep 17 15:43:44.581445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:44.581399    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" podStartSLOduration=1.5813851639999998 podStartE2EDuration="1.581385164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:43:44.579018613 +0000 UTC m=+816.546196434" watchObservedRunningTime="2026-09-17 15:43:44.581385164 +0000 UTC m=+816.548562979"
Sep 17 15:43:55.549105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:55.549074    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:43:55.549562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:43:55.549142    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:44:02.642961 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.642917    2564 generic.go:356] "Generic (PLEG): container finished" podID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerID="4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a" exitCode=0
Sep 17 15:44:02.643364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.642993    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerDied","Data":"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"}
Sep 17 15:44:02.643470 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.643453    2564 scope.go:117] "RemoveContainer" containerID="4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"
Sep 17 15:44:02.644738 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.644712    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:02.644794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.644780    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:02.644830 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:02.644808    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:03.651383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:03.651348    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerStarted","Data":"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"}
Sep 17 15:44:03.911722 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:03.911636    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:44:04.557088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:04.557057    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:04.557276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:04.557242    2564 patch_prober.go:28] interesting pod/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 15:44:04.557331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:04.557301    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 15:44:05.556786 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:05.556753    2564 patch_prober.go:28] interesting pod/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 15:44:05.557170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:05.556806    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 15:44:06.041284 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.041250    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"]
Sep 17 15:44:06.043975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.043958    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:06.046253 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.046232    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:44:06.046455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.046441    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:44:06.056856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.056834    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"]
Sep 17 15:44:06.148278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.148244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92\" (UID: \"5b2e43bc-88df-4d67-b793-a244a60f4cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:06.249909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.249860    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92\" (UID: \"5b2e43bc-88df-4d67-b793-a244a60f4cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:06.250519 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.250499    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92\" (UID: \"5b2e43bc-88df-4d67-b793-a244a60f4cd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:06.353836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.353755    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:06.557132 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.557099    2564 patch_prober.go:28] interesting pod/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 15:44:06.557534 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.557158    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 15:44:06.650012 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.649982    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"]
Sep 17 15:44:06.650878 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:44:06.650847    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2e43bc_88df_4d67_b793_a244a60f4cd6.slice/crio-3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a WatchSource:0}: Error finding container 3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a: Status 404 returned error can't find the container with id 3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a
Sep 17 15:44:06.669858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:06.669830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92" event={"ID":"5b2e43bc-88df-4d67-b793-a244a60f4cd6","Type":"ContainerStarted","Data":"3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a"}
Sep 17 15:44:07.556565 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:07.556526    2564 patch_prober.go:28] interesting pod/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 15:44:07.556774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:07.556582    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 15:44:07.676803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:07.676765    2564 generic.go:356] "Generic (PLEG): container finished" podID="5b2e43bc-88df-4d67-b793-a244a60f4cd6" containerID="919f3ebf8dc75b9e4f26e507ae5d3e14a401cf7f40905570edb257a1186c272a" exitCode=0
Sep 17 15:44:07.677171 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:07.676846    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92" event={"ID":"5b2e43bc-88df-4d67-b793-a244a60f4cd6","Type":"ContainerDied","Data":"919f3ebf8dc75b9e4f26e507ae5d3e14a401cf7f40905570edb257a1186c272a"}
Sep 17 15:44:08.556657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:08.556607    2564 patch_prober.go:28] interesting pod/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 17 15:44:08.556828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:08.556685    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 17 15:44:08.863438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:08.863412    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:08.980952 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:08.980914    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca\") pod \"5b2e43bc-88df-4d67-b793-a244a60f4cd6\" (UID: \"5b2e43bc-88df-4d67-b793-a244a60f4cd6\") "
Sep 17 15:44:08.981297 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:08.981270    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b2e43bc-88df-4d67-b793-a244a60f4cd6" (UID: "5b2e43bc-88df-4d67-b793-a244a60f4cd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:09.082579 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.082536    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b2e43bc-88df-4d67-b793-a244a60f4cd6-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:09.561386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.561359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:09.561812 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.561659    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:09.562243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.562227    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:09.688693 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.688645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92" event={"ID":"5b2e43bc-88df-4d67-b793-a244a60f4cd6","Type":"ContainerDied","Data":"3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a"}
Sep 17 15:44:09.688693 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.688676    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"
Sep 17 15:44:09.688693 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:09.688698    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3431981112e0fcde20e678f05715128fb3d6ee43a84a028bcadf4d2c52c8d90a"
Sep 17 15:44:12.368270 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.368235    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:44:12.368779 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.368438    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" podUID="b926977d-d35d-4fcb-9d0d-1700e558f91b" containerName="payload-processing" containerID="cri-o://4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3" gracePeriod=30
Sep 17 15:44:12.373739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.373714    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:44:12.373934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.373906    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" podUID="36d8e62f-b95b-489f-bc35-3180107c6fa9" containerName="payload-pre-processing" containerID="cri-o://3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc" gracePeriod=30
Sep 17 15:44:12.599856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.599829    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"]
Sep 17 15:44:12.603220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.603191    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-49b90ab7-cpt92"]
Sep 17 15:44:12.667817 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.667493    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:44:12.677092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.677006    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:44:12.716272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.715781    2564 generic.go:356] "Generic (PLEG): container finished" podID="b926977d-d35d-4fcb-9d0d-1700e558f91b" containerID="4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3" exitCode=0
Sep 17 15:44:12.716272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.715971    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" event={"ID":"b926977d-d35d-4fcb-9d0d-1700e558f91b","Type":"ContainerDied","Data":"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"}
Sep 17 15:44:12.716272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.716025    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r" event={"ID":"b926977d-d35d-4fcb-9d0d-1700e558f91b","Type":"ContainerDied","Data":"bd7a794c38196dfd0f8a80eb51df7384658357be741da8e4d3b3252cc08f1e37"}
Sep 17 15:44:12.716272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.716057    2564 scope.go:117] "RemoveContainer" containerID="4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"
Sep 17 15:44:12.716272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.716218    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"
Sep 17 15:44:12.719853 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.719804    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume\") pod \"b926977d-d35d-4fcb-9d0d-1700e558f91b\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") "
Sep 17 15:44:12.719957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.719886    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume\") pod \"36d8e62f-b95b-489f-bc35-3180107c6fa9\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") "
Sep 17 15:44:12.720027 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.720007    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2zhg\" (UniqueName: \"kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg\") pod \"36d8e62f-b95b-489f-bc35-3180107c6fa9\" (UID: \"36d8e62f-b95b-489f-bc35-3180107c6fa9\") "
Sep 17 15:44:12.720155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.720141    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvw65\" (UniqueName: \"kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65\") pod \"b926977d-d35d-4fcb-9d0d-1700e558f91b\" (UID: \"b926977d-d35d-4fcb-9d0d-1700e558f91b\") "
Sep 17 15:44:12.739550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.722049    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36d8e62f-b95b-489f-bc35-3180107c6fa9" (UID: "36d8e62f-b95b-489f-bc35-3180107c6fa9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:12.739550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.722399    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b926977d-d35d-4fcb-9d0d-1700e558f91b" (UID: "b926977d-d35d-4fcb-9d0d-1700e558f91b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:12.739550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.732512    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65" (OuterVolumeSpecName: "kube-api-access-xvw65") pod "b926977d-d35d-4fcb-9d0d-1700e558f91b" (UID: "b926977d-d35d-4fcb-9d0d-1700e558f91b"). InnerVolumeSpecName "kube-api-access-xvw65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:12.739550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.739080    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg" (OuterVolumeSpecName: "kube-api-access-j2zhg") pod "36d8e62f-b95b-489f-bc35-3180107c6fa9" (UID: "36d8e62f-b95b-489f-bc35-3180107c6fa9"). InnerVolumeSpecName "kube-api-access-j2zhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:12.742082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.742052    2564 generic.go:356] "Generic (PLEG): container finished" podID="36d8e62f-b95b-489f-bc35-3180107c6fa9" containerID="3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc" exitCode=0
Sep 17 15:44:12.742285 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.742267    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" event={"ID":"36d8e62f-b95b-489f-bc35-3180107c6fa9","Type":"ContainerDied","Data":"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"}
Sep 17 15:44:12.742516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.742496    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"
Sep 17 15:44:12.742658 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.742641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k" event={"ID":"36d8e62f-b95b-489f-bc35-3180107c6fa9","Type":"ContainerDied","Data":"e19ab1d3158137ddd51935a847ee4a9e93beca7b6519ad8c9bf087654e86bdf5"}
Sep 17 15:44:12.751578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.751556    2564 scope.go:117] "RemoveContainer" containerID="4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"
Sep 17 15:44:12.752994 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:12.752969    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3\": container with ID starting with 4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3 not found: ID does not exist" containerID="4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"
Sep 17 15:44:12.753698 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.753672    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3"} err="failed to get container status \"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3\": rpc error: code = NotFound desc = could not find container \"4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3\": container with ID starting with 4ba0313b0e3cceb1f0a26a8a51c8a02a339951cd7aa70bec632877c4fdbd40a3 not found: ID does not exist"
Sep 17 15:44:12.753853 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.753835    2564 scope.go:117] "RemoveContainer" containerID="3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"
Sep 17 15:44:12.754784 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.754767    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:44:12.755001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.754976    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy" containerID="cri-o://6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e" gracePeriod=30
Sep 17 15:44:12.765915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.765885    2564 scope.go:117] "RemoveContainer" containerID="3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"
Sep 17 15:44:12.766285 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:12.766261    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc\": container with ID starting with 3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc not found: ID does not exist" containerID="3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"
Sep 17 15:44:12.766449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.766424    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc"} err="failed to get container status \"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc\": rpc error: code = NotFound desc = could not find container \"3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc\": container with ID starting with 3b67dbd40ec72672068da3b570422291fe374e7f784e4a449cce3f52cde3dfdc not found: ID does not exist"
Sep 17 15:44:12.770834 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.770812    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:44:12.775389 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.775369    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-49b90ab7-577ff9545d-h762k"]
Sep 17 15:44:12.822204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.822178    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvw65\" (UniqueName: \"kubernetes.io/projected/b926977d-d35d-4fcb-9d0d-1700e558f91b-kube-api-access-xvw65\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.822204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.822201    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b926977d-d35d-4fcb-9d0d-1700e558f91b-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.822332 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.822210    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36d8e62f-b95b-489f-bc35-3180107c6fa9-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:12.822332 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:12.822219    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zhg\" (UniqueName: \"kubernetes.io/projected/36d8e62f-b95b-489f-bc35-3180107c6fa9-kube-api-access-j2zhg\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:13.039042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.039017    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:44:13.044565 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.044543    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-49b90ab7-7675f79c54-2ln4r"]
Sep 17 15:44:13.570857 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.570829    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:44:13.571274 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571262    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b926977d-d35d-4fcb-9d0d-1700e558f91b" containerName="payload-processing"
Sep 17 15:44:13.571318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571275    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b926977d-d35d-4fcb-9d0d-1700e558f91b" containerName="payload-processing"
Sep 17 15:44:13.571318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571287    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b2e43bc-88df-4d67-b793-a244a60f4cd6" containerName="revoke-keys"
Sep 17 15:44:13.571318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571292    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b2e43bc-88df-4d67-b793-a244a60f4cd6" containerName="revoke-keys"
Sep 17 15:44:13.571318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571300    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36d8e62f-b95b-489f-bc35-3180107c6fa9" containerName="payload-pre-processing"
Sep 17 15:44:13.571318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571306    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="36d8e62f-b95b-489f-bc35-3180107c6fa9" containerName="payload-pre-processing"
Sep 17 15:44:13.571472 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571368    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b926977d-d35d-4fcb-9d0d-1700e558f91b" containerName="payload-processing"
Sep 17 15:44:13.571472 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571376    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="36d8e62f-b95b-489f-bc35-3180107c6fa9" containerName="payload-pre-processing"
Sep 17 15:44:13.571472 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.571383    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b2e43bc-88df-4d67-b793-a244a60f4cd6" containerName="revoke-keys"
Sep 17 15:44:13.574362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.574345    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.578031 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.578012    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6b2abe85-openshift-default-dockercfg-fd2q6\""
Sep 17 15:44:13.590549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.590526    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:44:13.629274 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629248    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629392 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629286    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629392 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629305    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629414    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629443    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629479    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wl7z\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629523    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629678 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629609    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.629734 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.629685    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730286    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730322    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730362    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730379    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730405    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wl7z\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730438    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730479    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730513    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730584    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730964 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730783    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.730964 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.730831    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.731077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.731058    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.731139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.731083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.731366 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.731339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.732827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.732804    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.733029 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.733013    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.744924 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.744899    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.746230 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.746188    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wl7z\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z\") pod \"e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") " pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:13.884253 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:13.884225    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:14.172774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.172750    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:44:14.174549 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:44:14.174522    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3e7db46_3b2c_4954_a97b_29e9d7b51192.slice/crio-31e9c528ce9aabc85613b5a0c291770c4e095d5ce6f794c66cd57180731218bb WatchSource:0}: Error finding container 31e9c528ce9aabc85613b5a0c291770c4e095d5ce6f794c66cd57180731218bb: Status 404 returned error can't find the container with id 31e9c528ce9aabc85613b5a0c291770c4e095d5ce6f794c66cd57180731218bb
Sep 17 15:44:14.176833 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.176802    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:14.176900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.176872    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:14.176939 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.176898    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:14.526062 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.526031    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36d8e62f-b95b-489f-bc35-3180107c6fa9" path="/var/lib/kubelet/pods/36d8e62f-b95b-489f-bc35-3180107c6fa9/volumes"
Sep 17 15:44:14.526417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.526403    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b2e43bc-88df-4d67-b793-a244a60f4cd6" path="/var/lib/kubelet/pods/5b2e43bc-88df-4d67-b793-a244a60f4cd6/volumes"
Sep 17 15:44:14.526754 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.526740    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b926977d-d35d-4fcb-9d0d-1700e558f91b" path="/var/lib/kubelet/pods/b926977d-d35d-4fcb-9d0d-1700e558f91b/volumes"
Sep 17 15:44:14.766812 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.766767    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerStarted","Data":"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"}
Sep 17 15:44:14.767274 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.766841    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerStarted","Data":"31e9c528ce9aabc85613b5a0c291770c4e095d5ce6f794c66cd57180731218bb"}
Sep 17 15:44:14.791519 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.791398    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podStartSLOduration=1.7913751549999999 podStartE2EDuration="1.791375155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:14.790015711 +0000 UTC m=+846.757193529" watchObservedRunningTime="2026-09-17 15:44:14.791375155 +0000 UTC m=+846.758552974"
Sep 17 15:44:14.884698 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:14.884667    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:15.890358 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:15.890332    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:16.785708 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:16.785671    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:16.786683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:16.786660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:18.024454 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.024389    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:18.179201 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179170    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179223    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179252    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179285    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179379 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179450    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb7xl\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179455    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:44:18.179582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179500    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179552    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179581    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data\") pod \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\" (UID: \"1375ddd9-ab6d-42f1-9a89-c6502dd93d98\") "
Sep 17 15:44:18.179847 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179641    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:44:18.179847 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:44:18.179970 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179948    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data" (OuterVolumeSpecName: "istio-data") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:44:18.180023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179998    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.180023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.179997    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:18.180096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.180017    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.180096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.180060    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.181629 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.181590    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl" (OuterVolumeSpecName: "kube-api-access-cb7xl") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "kube-api-access-cb7xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:18.181742 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.181639    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:44:18.181863 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.181846    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token" (OuterVolumeSpecName: "istio-token") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:18.181950 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.181929    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1375ddd9-ab6d-42f1-9a89-c6502dd93d98" (UID: "1375ddd9-ab6d-42f1-9a89-c6502dd93d98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281383    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb7xl\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-kube-api-access-cb7xl\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281414    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281424    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281432    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281439    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.281468 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.281447    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1375ddd9-ab6d-42f1-9a89-c6502dd93d98-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:18.798545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.798465    2564 generic.go:356] "Generic (PLEG): container finished" podID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerID="6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e" exitCode=0
Sep 17 15:44:18.798545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.798513    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"
Sep 17 15:44:18.798769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.798542    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerDied","Data":"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"}
Sep 17 15:44:18.798769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.798590    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r" event={"ID":"1375ddd9-ab6d-42f1-9a89-c6502dd93d98","Type":"ContainerDied","Data":"d69570919d6bf0f0e3976215cbd2d4c1a2559c2e1f69fb67e64ad5c304520cbe"}
Sep 17 15:44:18.798769 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.798625    2564 scope.go:117] "RemoveContainer" containerID="6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"
Sep 17 15:44:18.807704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.807687    2564 scope.go:117] "RemoveContainer" containerID="4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"
Sep 17 15:44:18.816406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.816382    2564 scope.go:117] "RemoveContainer" containerID="6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"
Sep 17 15:44:18.816696 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:18.816676    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e\": container with ID starting with 6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e not found: ID does not exist" containerID="6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"
Sep 17 15:44:18.816799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.816703    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e"} err="failed to get container status \"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e\": rpc error: code = NotFound desc = could not find container \"6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e\": container with ID starting with 6c7df91b337d88d6933d3ef35d8b621ce5d11b7e0fcf555afa3e93782687934e not found: ID does not exist"
Sep 17 15:44:18.816799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.816721    2564 scope.go:117] "RemoveContainer" containerID="4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"
Sep 17 15:44:18.817026 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:18.817004    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a\": container with ID starting with 4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a not found: ID does not exist" containerID="4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"
Sep 17 15:44:18.817087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.817031    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a"} err="failed to get container status \"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a\": rpc error: code = NotFound desc = could not find container \"4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a\": container with ID starting with 4f7113dad3c554159deebaa0c194a57f770a946fbf52cdecc094533ca181203a not found: ID does not exist"
Sep 17 15:44:18.817591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.817569    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:44:18.822366 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:18.822345    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-49b90ab7-gw-openshift-default-65bc6f64dd-fjg7r"]
Sep 17 15:44:19.913199 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913168    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:19.913771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913754    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.913771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913774    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.913884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913784    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.913884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913791    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.913884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913873    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.913884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.913885    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" containerName="istio-proxy"
Sep 17 15:44:19.916876 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.916859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:19.919457 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.919438    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6b2abe85\""
Sep 17 15:44:19.935142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.935119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:19.938086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.938061    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:19.938210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.938175    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:19.940548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.940526    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6b2abe85-dockercfg-wg6xn\""
Sep 17 15:44:19.949251 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.949231    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:19.999774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.999748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ftb\" (UniqueName: \"kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:19.999906 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:19.999887    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.101409 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.101371    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99fs6\" (UniqueName: \"kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.101569 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.101429    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.101569 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.101476    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.101719 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.101578    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ftb\" (UniqueName: \"kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.102179 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.102162    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.110514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.110494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ftb\" (UniqueName: \"kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb\") pod \"payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.202599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.202504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99fs6\" (UniqueName: \"kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.202599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.202561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.203276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.203257    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.212413 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.212382    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99fs6\" (UniqueName: \"kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6\") pod \"payload-processing-e2e-ait-6b2abe85-b86f59889-5j527\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") " pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.227048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.227025    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.248261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.248239    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.501170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.501148    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:20.502775 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:44:20.502751    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb654c0d6_eac3_45ef_a8c6_46adda78ba0a.slice/crio-effbf85be64cf60968fbea0ec4c32bdd35a65206aa5a3937f7382c760daad7ad WatchSource:0}: Error finding container effbf85be64cf60968fbea0ec4c32bdd35a65206aa5a3937f7382c760daad7ad: Status 404 returned error can't find the container with id effbf85be64cf60968fbea0ec4c32bdd35a65206aa5a3937f7382c760daad7ad
Sep 17 15:44:20.528375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.528349    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1375ddd9-ab6d-42f1-9a89-c6502dd93d98" path="/var/lib/kubelet/pods/1375ddd9-ab6d-42f1-9a89-c6502dd93d98/volumes"
Sep 17 15:44:20.534503 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.534482    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:20.536168 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:44:20.536147    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode110e815_2279_4b19_9fb2_1016627d755f.slice/crio-d9cc62b5f811c32741e3670ad724a51bafa4bd83097599a99ad39cb9cd2c1db9 WatchSource:0}: Error finding container d9cc62b5f811c32741e3670ad724a51bafa4bd83097599a99ad39cb9cd2c1db9: Status 404 returned error can't find the container with id d9cc62b5f811c32741e3670ad724a51bafa4bd83097599a99ad39cb9cd2c1db9
Sep 17 15:44:20.815011 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.814974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" event={"ID":"e110e815-2279-4b19-9fb2-1016627d755f","Type":"ContainerStarted","Data":"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"}
Sep 17 15:44:20.815222 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.815035    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" event={"ID":"e110e815-2279-4b19-9fb2-1016627d755f","Type":"ContainerStarted","Data":"d9cc62b5f811c32741e3670ad724a51bafa4bd83097599a99ad39cb9cd2c1db9"}
Sep 17 15:44:20.815222 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.815062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:20.816935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.816908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" event={"ID":"b654c0d6-eac3-45ef-a8c6-46adda78ba0a","Type":"ContainerStarted","Data":"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"}
Sep 17 15:44:20.817077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.816958    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" event={"ID":"b654c0d6-eac3-45ef-a8c6-46adda78ba0a","Type":"ContainerStarted","Data":"effbf85be64cf60968fbea0ec4c32bdd35a65206aa5a3937f7382c760daad7ad"}
Sep 17 15:44:20.817077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.817000    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:20.833280 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.833225    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" podStartSLOduration=1.833207963 podStartE2EDuration="1.833207963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:20.832559008 +0000 UTC m=+852.799736825" watchObservedRunningTime="2026-09-17 15:44:20.833207963 +0000 UTC m=+852.800385775"
Sep 17 15:44:20.854048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:20.853990    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" podStartSLOduration=1.853970664 podStartE2EDuration="1.853970664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:44:20.850545624 +0000 UTC m=+852.817723443" watchObservedRunningTime="2026-09-17 15:44:20.853970664 +0000 UTC m=+852.821148485"
Sep 17 15:44:23.596159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:23.596124    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:44:31.823837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:31.823803    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:31.824422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:31.823976    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:36.302661 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.302626    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"]
Sep 17 15:44:36.306293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.306273    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:36.309350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.309330    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:44:36.309463 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.309421    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:44:36.323228 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.323209    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"]
Sep 17 15:44:36.364295 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.364269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-28eba6-mwr2c\" (UID: \"c8a62a9d-cffd-493c-9cb4-52ced1092765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:36.465174 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.465140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-28eba6-mwr2c\" (UID: \"c8a62a9d-cffd-493c-9cb4-52ced1092765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:36.465806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.465782    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-28eba6-mwr2c\" (UID: \"c8a62a9d-cffd-493c-9cb4-52ced1092765\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:36.618168 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.618083    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:36.941435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:36.941380    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"]
Sep 17 15:44:37.951032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:37.951001    2564 generic.go:356] "Generic (PLEG): container finished" podID="c8a62a9d-cffd-493c-9cb4-52ced1092765" containerID="b5ae3bb3b61f0305b440c7ccc52c9a3fbe60fd3db0b59b78d856eaec114155fa" exitCode=0
Sep 17 15:44:37.951323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:37.951078    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c" event={"ID":"c8a62a9d-cffd-493c-9cb4-52ced1092765","Type":"ContainerDied","Data":"b5ae3bb3b61f0305b440c7ccc52c9a3fbe60fd3db0b59b78d856eaec114155fa"}
Sep 17 15:44:37.951323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:37.951127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c" event={"ID":"c8a62a9d-cffd-493c-9cb4-52ced1092765","Type":"ContainerStarted","Data":"c30fcf4a6963ce1e55cd72f0c3e332da87d7db0e75b8f8d7922d9372c0e01ff7"}
Sep 17 15:44:39.100254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.100228    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:39.193359 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.193325    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca\") pod \"c8a62a9d-cffd-493c-9cb4-52ced1092765\" (UID: \"c8a62a9d-cffd-493c-9cb4-52ced1092765\") "
Sep 17 15:44:39.193716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.193695    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8a62a9d-cffd-493c-9cb4-52ced1092765" (UID: "c8a62a9d-cffd-493c-9cb4-52ced1092765"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:39.294823 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.294788    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8a62a9d-cffd-493c-9cb4-52ced1092765-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:39.963706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.963669    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"
Sep 17 15:44:39.963706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.963694    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c" event={"ID":"c8a62a9d-cffd-493c-9cb4-52ced1092765","Type":"ContainerDied","Data":"c30fcf4a6963ce1e55cd72f0c3e332da87d7db0e75b8f8d7922d9372c0e01ff7"}
Sep 17 15:44:39.963921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:39.963737    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c30fcf4a6963ce1e55cd72f0c3e332da87d7db0e75b8f8d7922d9372c0e01ff7"
Sep 17 15:44:45.998409 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:45.998376    2564 generic.go:356] "Generic (PLEG): container finished" podID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerID="fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d" exitCode=0
Sep 17 15:44:45.998870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:45.998451    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerDied","Data":"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"}
Sep 17 15:44:45.998870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:45.998867    2564 scope.go:117] "RemoveContainer" containerID="fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"
Sep 17 15:44:46.000063 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:46.000038    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:46.000129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:46.000101    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:46.000166 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:46.000135    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:44:46.251455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:46.251357    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:44:46.785864 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:46.785830    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:47.006830 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.006792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerStarted","Data":"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"}
Sep 17 15:44:47.121692 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.115676    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"]
Sep 17 15:44:47.121692 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.116817    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a62a9d-cffd-493c-9cb4-52ced1092765" containerName="revoke-keys"
Sep 17 15:44:47.121692 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.116837    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a62a9d-cffd-493c-9cb4-52ced1092765" containerName="revoke-keys"
Sep 17 15:44:47.121692 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.117509    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a62a9d-cffd-493c-9cb4-52ced1092765" containerName="revoke-keys"
Sep 17 15:44:47.127406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.127383    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:47.130408 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.130382    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:44:47.134906 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.134885    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:44:47.143527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.143247    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"]
Sep 17 15:44:47.191582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.191555    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj\" (UID: \"e5ce718f-b6ff-4758-bcff-f7f92b2305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:47.292593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.292556    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj\" (UID: \"e5ce718f-b6ff-4758-bcff-f7f92b2305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:47.293163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.293143    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj\" (UID: \"e5ce718f-b6ff-4758-bcff-f7f92b2305f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:47.449853 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.449768    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:47.759821 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.759788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"]
Sep 17 15:44:47.766141 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:44:47.766110    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ce718f_b6ff_4758_bcff_f7f92b2305f9.slice/crio-36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f WatchSource:0}: Error finding container 36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f: Status 404 returned error can't find the container with id 36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f
Sep 17 15:44:47.885423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.885389    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:47.886509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.886482    2564 patch_prober.go:28] interesting pod/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 17 15:44:47.886662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:47.886562    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 17 15:44:48.018323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:48.018229    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj" event={"ID":"e5ce718f-b6ff-4758-bcff-f7f92b2305f9","Type":"ContainerStarted","Data":"36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f"}
Sep 17 15:44:48.885084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:48.885048    2564 patch_prober.go:28] interesting pod/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 17 15:44:48.885264 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:48.885106    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 17 15:44:49.046867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.046826    2564 generic.go:356] "Generic (PLEG): container finished" podID="e5ce718f-b6ff-4758-bcff-f7f92b2305f9" containerID="2cf738ad3f041f2b6bc0b0b5448174b02398bd67b76e70a6db69de037cc688ab" exitCode=0
Sep 17 15:44:49.047368 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.046869    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj" event={"ID":"e5ce718f-b6ff-4758-bcff-f7f92b2305f9","Type":"ContainerDied","Data":"2cf738ad3f041f2b6bc0b0b5448174b02398bd67b76e70a6db69de037cc688ab"}
Sep 17 15:44:49.449736 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.449564    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"]
Sep 17 15:44:49.453088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.453057    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-28eba6-mwr2c"]
Sep 17 15:44:49.885088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.885058    2564 patch_prober.go:28] interesting pod/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 17 15:44:49.885260 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:49.885114    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 17 15:44:50.232244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.232209    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:50.340112 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.340076    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca\") pod \"e5ce718f-b6ff-4758-bcff-f7f92b2305f9\" (UID: \"e5ce718f-b6ff-4758-bcff-f7f92b2305f9\") "
Sep 17 15:44:50.340429 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.340407    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5ce718f-b6ff-4758-bcff-f7f92b2305f9" (UID: "e5ce718f-b6ff-4758-bcff-f7f92b2305f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:50.441825 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.441745    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5ce718f-b6ff-4758-bcff-f7f92b2305f9-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:50.529818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.529777    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a62a9d-cffd-493c-9cb4-52ced1092765" path="/var/lib/kubelet/pods/c8a62a9d-cffd-493c-9cb4-52ced1092765/volumes"
Sep 17 15:44:50.885558 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.885526    2564 patch_prober.go:28] interesting pod/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 17 15:44:50.885755 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:50.885578    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 17 15:44:51.060388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:51.060362    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"
Sep 17 15:44:51.060547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:51.060389    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj" event={"ID":"e5ce718f-b6ff-4758-bcff-f7f92b2305f9","Type":"ContainerDied","Data":"36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f"}
Sep 17 15:44:51.060547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:51.060440    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36e85be265930c8b43a3f8d39ad381d8c5523b10e4512c7ff36e21fbad99a58f"
Sep 17 15:44:51.885294 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:51.885257    2564 patch_prober.go:28] interesting pod/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Sep 17 15:44:51.885688 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:51.885311    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Sep 17 15:44:52.889217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:52.889189    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:52.889574 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:52.889473    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:52.890194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:52.890176    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:44:58.172348 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.172318    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:58.173395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.173337    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" podUID="e110e815-2279-4b19-9fb2-1016627d755f" containerName="payload-processing" containerID="cri-o://7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1" gracePeriod=30
Sep 17 15:44:58.184518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.184015    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:58.184518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.184286    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" podUID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" containerName="payload-pre-processing" containerID="cri-o://5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60" gracePeriod=30
Sep 17 15:44:58.360882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.360849    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"]
Sep 17 15:44:58.365553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.365531    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6b2abe85-jndnj"]
Sep 17 15:44:58.463879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.463855    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:58.466845 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.466823    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:58.560516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.560270    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume\") pod \"e110e815-2279-4b19-9fb2-1016627d755f\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") "
Sep 17 15:44:58.560516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.560352    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume\") pod \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") "
Sep 17 15:44:58.560516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.560388    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99fs6\" (UniqueName: \"kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6\") pod \"e110e815-2279-4b19-9fb2-1016627d755f\" (UID: \"e110e815-2279-4b19-9fb2-1016627d755f\") "
Sep 17 15:44:58.560516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.560524    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45ftb\" (UniqueName: \"kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb\") pod \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\" (UID: \"b654c0d6-eac3-45ef-a8c6-46adda78ba0a\") "
Sep 17 15:44:58.561575 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.561546    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b654c0d6-eac3-45ef-a8c6-46adda78ba0a" (UID: "b654c0d6-eac3-45ef-a8c6-46adda78ba0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:58.561772 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.561749    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e110e815-2279-4b19-9fb2-1016627d755f" (UID: "e110e815-2279-4b19-9fb2-1016627d755f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:44:58.563576 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.563539    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb" (OuterVolumeSpecName: "kube-api-access-45ftb") pod "b654c0d6-eac3-45ef-a8c6-46adda78ba0a" (UID: "b654c0d6-eac3-45ef-a8c6-46adda78ba0a"). InnerVolumeSpecName "kube-api-access-45ftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:58.563955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.563922    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6" (OuterVolumeSpecName: "kube-api-access-99fs6") pod "e110e815-2279-4b19-9fb2-1016627d755f" (UID: "e110e815-2279-4b19-9fb2-1016627d755f"). InnerVolumeSpecName "kube-api-access-99fs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:44:58.565115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.565091    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5ce718f-b6ff-4758-bcff-f7f92b2305f9" path="/var/lib/kubelet/pods/e5ce718f-b6ff-4758-bcff-f7f92b2305f9/volumes"
Sep 17 15:44:58.661882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.661791    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e110e815-2279-4b19-9fb2-1016627d755f-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.661882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.661828    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.661882 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.661842    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99fs6\" (UniqueName: \"kubernetes.io/projected/e110e815-2279-4b19-9fb2-1016627d755f-kube-api-access-99fs6\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:58.662237 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:58.662053    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45ftb\" (UniqueName: \"kubernetes.io/projected/b654c0d6-eac3-45ef-a8c6-46adda78ba0a-kube-api-access-45ftb\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:44:59.112458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.112423    2564 generic.go:356] "Generic (PLEG): container finished" podID="e110e815-2279-4b19-9fb2-1016627d755f" containerID="7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1" exitCode=0
Sep 17 15:44:59.112704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.112499    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"
Sep 17 15:44:59.112704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.112508    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" event={"ID":"e110e815-2279-4b19-9fb2-1016627d755f","Type":"ContainerDied","Data":"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"}
Sep 17 15:44:59.112704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.112578    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527" event={"ID":"e110e815-2279-4b19-9fb2-1016627d755f","Type":"ContainerDied","Data":"d9cc62b5f811c32741e3670ad724a51bafa4bd83097599a99ad39cb9cd2c1db9"}
Sep 17 15:44:59.112704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.112629    2564 scope.go:117] "RemoveContainer" containerID="7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"
Sep 17 15:44:59.114018 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.113997    2564 generic.go:356] "Generic (PLEG): container finished" podID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" containerID="5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60" exitCode=0
Sep 17 15:44:59.114150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.114048    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"
Sep 17 15:44:59.114150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.114082    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" event={"ID":"b654c0d6-eac3-45ef-a8c6-46adda78ba0a","Type":"ContainerDied","Data":"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"}
Sep 17 15:44:59.114150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.114146    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x" event={"ID":"b654c0d6-eac3-45ef-a8c6-46adda78ba0a","Type":"ContainerDied","Data":"effbf85be64cf60968fbea0ec4c32bdd35a65206aa5a3937f7382c760daad7ad"}
Sep 17 15:44:59.122261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.122246    2564 scope.go:117] "RemoveContainer" containerID="7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"
Sep 17 15:44:59.122536 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:59.122520    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1\": container with ID starting with 7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1 not found: ID does not exist" containerID="7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"
Sep 17 15:44:59.122587 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.122544    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1"} err="failed to get container status \"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1\": rpc error: code = NotFound desc = could not find container \"7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1\": container with ID starting with 7180f53b73b79e700f909be1d9f9075ff85a16e5cb0d40a877a7d233d464b2b1 not found: ID does not exist"
Sep 17 15:44:59.122587 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.122558    2564 scope.go:117] "RemoveContainer" containerID="5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"
Sep 17 15:44:59.130596 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.130578    2564 scope.go:117] "RemoveContainer" containerID="5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"
Sep 17 15:44:59.130880 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:44:59.130862    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60\": container with ID starting with 5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60 not found: ID does not exist" containerID="5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"
Sep 17 15:44:59.130940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.130887    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60"} err="failed to get container status \"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60\": rpc error: code = NotFound desc = could not find container \"5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60\": container with ID starting with 5568fe2144face89832afc366d7be9773dba5cb8b4d318e3624aee606ac90d60 not found: ID does not exist"
Sep 17 15:44:59.140654 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.140630    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:59.148574 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.148546    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6b2abe85-5b959dffcd-rp26x"]
Sep 17 15:44:59.202369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.202343    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:59.212277 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.212253    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6b2abe85-b86f59889-5j527"]
Sep 17 15:44:59.538990 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.538958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:44:59.539234 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:44:59.539209    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy" containerID="cri-o://3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c" gracePeriod=30
Sep 17 15:45:00.167418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167386    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"]
Sep 17 15:45:00.167841 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167826    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" containerName="payload-pre-processing"
Sep 17 15:45:00.167841 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167842    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" containerName="payload-pre-processing"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167852    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5ce718f-b6ff-4758-bcff-f7f92b2305f9" containerName="revoke-keys"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167858    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5ce718f-b6ff-4758-bcff-f7f92b2305f9" containerName="revoke-keys"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167881    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e110e815-2279-4b19-9fb2-1016627d755f" containerName="payload-processing"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167886    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e110e815-2279-4b19-9fb2-1016627d755f" containerName="payload-processing"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167962    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" containerName="payload-pre-processing"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167970    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5ce718f-b6ff-4758-bcff-f7f92b2305f9" containerName="revoke-keys"
Sep 17 15:45:00.167989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.167979    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e110e815-2279-4b19-9fb2-1016627d755f" containerName="payload-processing"
Sep 17 15:45:00.171146 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.171119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"]
Sep 17 15:45:00.171279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.171266    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"
Sep 17 15:45:00.173487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.173468    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:45:00.173665 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.173649    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"]
Sep 17 15:45:00.173726 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.173678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"
Sep 17 15:45:00.175676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.175654    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"
Sep 17 15:45:00.180597 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.180572    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"]
Sep 17 15:45:00.185364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.185184    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"
Sep 17 15:45:00.193002 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.192964    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"]
Sep 17 15:45:00.199209 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.197237    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"
Sep 17 15:45:00.214928 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.214884    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"]
Sep 17 15:45:00.215448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.215429    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"]
Sep 17 15:45:00.215599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.215321    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"
Sep 17 15:45:00.223912 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.223256    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"]
Sep 17 15:45:00.255278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.255245    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"
Sep 17 15:45:00.264490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.264464    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"
Sep 17 15:45:00.339644 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.337512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:00.341457 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.341418    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.348205 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.346219    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-23da8b90-openshift-default-dockercfg-klhxm\""
Sep 17 15:45:00.358855 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.358783    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:00.381330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381298    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381337    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381357    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381699 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381442    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381699 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381482    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381699 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381533    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381699 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381588    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.381699 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg2gq\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.382075 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.381815    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.482744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.482671    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.482888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.482773    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.482888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.482880    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg2gq\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.482975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.482917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.482974    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483002    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483027    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483083    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483196    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483247    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483367    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.483724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483653    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.484032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.483884    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.484128 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.484105    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.486243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.486222    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.486860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.486839    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.496509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.496486    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.496771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.496748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg2gq\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq\") pod \"e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") " pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.509755 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.509731    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"]
Sep 17 15:45:00.512094 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:00.512069    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf545ab57_26f0_4b1c_8061_2c4e4a39fde2.slice/crio-28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4 WatchSource:0}: Error finding container 28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4: Status 404 returned error can't find the container with id 28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4
Sep 17 15:45:00.512684 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:00.512663    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161e5807_e23c_4597_80d5_67a1363f1186.slice/crio-f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328 WatchSource:0}: Error finding container f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328: Status 404 returned error can't find the container with id f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328
Sep 17 15:45:00.513107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.513085    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"]
Sep 17 15:45:00.526180 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.526150    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b654c0d6-eac3-45ef-a8c6-46adda78ba0a" path="/var/lib/kubelet/pods/b654c0d6-eac3-45ef-a8c6-46adda78ba0a/volumes"
Sep 17 15:45:00.526583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.526563    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e110e815-2279-4b19-9fb2-1016627d755f" path="/var/lib/kubelet/pods/e110e815-2279-4b19-9fb2-1016627d755f/volumes"
Sep 17 15:45:00.570815 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.570793    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"]
Sep 17 15:45:00.572681 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:00.572644    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f887bf0_c4df_4579_9126_7b87808584fe.slice/crio-fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722 WatchSource:0}: Error finding container fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722: Status 404 returned error can't find the container with id fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722
Sep 17 15:45:00.601628 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.601588    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"]
Sep 17 15:45:00.612980 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:00.612955    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded50bed4_416a_4012_86a5_8f072c0c4b4d.slice/crio-bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be WatchSource:0}: Error finding container bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be: Status 404 returned error can't find the container with id bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be
Sep 17 15:45:00.663012 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.662991    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:00.961338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.961313    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:00.963291 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:00.963271    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f962857_8ae0_481a_80fd_95755547031b.slice/crio-7e9943f1697a7455d2f0aedce746cb677a9ce4c38a2ba6d857e072545b52599c WatchSource:0}: Error finding container 7e9943f1697a7455d2f0aedce746cb677a9ce4c38a2ba6d857e072545b52599c: Status 404 returned error can't find the container with id 7e9943f1697a7455d2f0aedce746cb677a9ce4c38a2ba6d857e072545b52599c
Sep 17 15:45:00.965697 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.965664    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:00.965801 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.965742    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:00.965801 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:00.965784    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:01.130759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.130579    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerStarted","Data":"9debd76761f50fd90e1d2de7975a51da69d21caf458ccb90da0af41925bb8b1e"}
Sep 17 15:45:01.130759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.130663    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerStarted","Data":"7e9943f1697a7455d2f0aedce746cb677a9ce4c38a2ba6d857e072545b52599c"}
Sep 17 15:45:01.132480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.132448    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf" event={"ID":"1f887bf0-c4df-4579-9126-7b87808584fe","Type":"ContainerStarted","Data":"4dc5397100d33e99952f708cf77d0b7a834d7e6a446adaca3ba7f009a5d58d37"}
Sep 17 15:45:01.132663 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.132503    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf" event={"ID":"1f887bf0-c4df-4579-9126-7b87808584fe","Type":"ContainerStarted","Data":"fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722"}
Sep 17 15:45:01.134279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.134252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh" event={"ID":"f545ab57-26f0-4b1c-8061-2c4e4a39fde2","Type":"ContainerStarted","Data":"a334e7e3008226f2eff48be43ea3c587eb6007771bbdc0aa8610f7dbd50cc0ba"}
Sep 17 15:45:01.134386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.134302    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh" event={"ID":"f545ab57-26f0-4b1c-8061-2c4e4a39fde2","Type":"ContainerStarted","Data":"28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4"}
Sep 17 15:45:01.135948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.135920    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs" event={"ID":"ed50bed4-416a-4012-86a5-8f072c0c4b4d","Type":"ContainerStarted","Data":"901b87542f8f9ce6e91eaee4888a8561e50086171576b6d59addef39f496175d"}
Sep 17 15:45:01.136063 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.135974    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs" event={"ID":"ed50bed4-416a-4012-86a5-8f072c0c4b4d","Type":"ContainerStarted","Data":"bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be"}
Sep 17 15:45:01.137595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.137573    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl" event={"ID":"161e5807-e23c-4597-80d5-67a1363f1186","Type":"ContainerStarted","Data":"b05ffae186f709ff5c5c3085379f124d3ae6d99a8935472f315ff63b4827b210"}
Sep 17 15:45:01.137719 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.137622    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl" event={"ID":"161e5807-e23c-4597-80d5-67a1363f1186","Type":"ContainerStarted","Data":"f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328"}
Sep 17 15:45:01.166373 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.166321    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podStartSLOduration=1.166306186 podStartE2EDuration="1.166306186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:01.163509938 +0000 UTC m=+893.130687770" watchObservedRunningTime="2026-09-17 15:45:01.166306186 +0000 UTC m=+893.133484004"
Sep 17 15:45:01.188130 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.188080    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf" podStartSLOduration=1.188060106 podStartE2EDuration="1.188060106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:01.186544864 +0000 UTC m=+893.153722683" watchObservedRunningTime="2026-09-17 15:45:01.188060106 +0000 UTC m=+893.155237924"
Sep 17 15:45:01.205988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.205926    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl" podStartSLOduration=1.205901589 podStartE2EDuration="1.205901589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:01.203076788 +0000 UTC m=+893.170254606" watchObservedRunningTime="2026-09-17 15:45:01.205901589 +0000 UTC m=+893.173079408"
Sep 17 15:45:01.221853 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.221792    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh" podStartSLOduration=1.22177022 podStartE2EDuration="1.22177022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:01.22076313 +0000 UTC m=+893.187940962" watchObservedRunningTime="2026-09-17 15:45:01.22177022 +0000 UTC m=+893.188948069"
Sep 17 15:45:01.238038 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.237986    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs" podStartSLOduration=1.237971528 podStartE2EDuration="1.237971528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:01.236594012 +0000 UTC m=+893.203771841" watchObservedRunningTime="2026-09-17 15:45:01.237971528 +0000 UTC m=+893.205149345"
Sep 17 15:45:01.664101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:01.664012    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:02.145725 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.145692    2564 generic.go:356] "Generic (PLEG): container finished" podID="161e5807-e23c-4597-80d5-67a1363f1186" containerID="b05ffae186f709ff5c5c3085379f124d3ae6d99a8935472f315ff63b4827b210" exitCode=0
Sep 17 15:45:02.145906 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.145729    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl" event={"ID":"161e5807-e23c-4597-80d5-67a1363f1186","Type":"ContainerDied","Data":"b05ffae186f709ff5c5c3085379f124d3ae6d99a8935472f315ff63b4827b210"}
Sep 17 15:45:02.147342 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.147321    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f887bf0-c4df-4579-9126-7b87808584fe" containerID="4dc5397100d33e99952f708cf77d0b7a834d7e6a446adaca3ba7f009a5d58d37" exitCode=0
Sep 17 15:45:02.147436 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.147397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf" event={"ID":"1f887bf0-c4df-4579-9126-7b87808584fe","Type":"ContainerDied","Data":"4dc5397100d33e99952f708cf77d0b7a834d7e6a446adaca3ba7f009a5d58d37"}
Sep 17 15:45:02.148852 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.148832    2564 generic.go:356] "Generic (PLEG): container finished" podID="f545ab57-26f0-4b1c-8061-2c4e4a39fde2" containerID="a334e7e3008226f2eff48be43ea3c587eb6007771bbdc0aa8610f7dbd50cc0ba" exitCode=0
Sep 17 15:45:02.148959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.148908    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh" event={"ID":"f545ab57-26f0-4b1c-8061-2c4e4a39fde2","Type":"ContainerDied","Data":"a334e7e3008226f2eff48be43ea3c587eb6007771bbdc0aa8610f7dbd50cc0ba"}
Sep 17 15:45:02.150336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.150318    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed50bed4-416a-4012-86a5-8f072c0c4b4d" containerID="901b87542f8f9ce6e91eaee4888a8561e50086171576b6d59addef39f496175d" exitCode=0
Sep 17 15:45:02.150419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.150362    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs" event={"ID":"ed50bed4-416a-4012-86a5-8f072c0c4b4d","Type":"ContainerDied","Data":"901b87542f8f9ce6e91eaee4888a8561e50086171576b6d59addef39f496175d"}
Sep 17 15:45:02.669283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:02.669254    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:03.157213 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.157181    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:03.158834 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.158804    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:03.307373 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.307348    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"
Sep 17 15:45:03.364988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.364902    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"
Sep 17 15:45:03.395609 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.395588    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"
Sep 17 15:45:03.399324 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:03.399301    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"
Sep 17 15:45:04.163964 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.163934    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh"
Sep 17 15:45:04.163964 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.163946    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827665-fm2kh" event={"ID":"f545ab57-26f0-4b1c-8061-2c4e4a39fde2","Type":"ContainerDied","Data":"28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4"}
Sep 17 15:45:04.164444 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.163993    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28885aec8cfe59e30d2b763a48c4908278a76700ca9fc28161cce0ff57ec46a4"
Sep 17 15:45:04.165268 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.165247    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"
Sep 17 15:45:04.165391 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.165288    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs" event={"ID":"ed50bed4-416a-4012-86a5-8f072c0c4b4d","Type":"ContainerDied","Data":"bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be"}
Sep 17 15:45:04.165391 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.165339    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd1b04508232e8cb72dbb13e28e9a621c40a80014db3c3dda35af710122645be"
Sep 17 15:45:04.166759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.166736    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"
Sep 17 15:45:04.166866 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.166763    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl" event={"ID":"161e5807-e23c-4597-80d5-67a1363f1186","Type":"ContainerDied","Data":"f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328"}
Sep 17 15:45:04.166866 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.166807    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d1031b87ed5689d248aef9e35454775f8477e735fb7f73b1ee310349962328"
Sep 17 15:45:04.168226 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.168201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf" event={"ID":"1f887bf0-c4df-4579-9126-7b87808584fe","Type":"ContainerDied","Data":"fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722"}
Sep 17 15:45:04.168305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.168252    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1c95b7a964f475535055657120c8725351deaf486524c81319ecdcd8ed5722"
Sep 17 15:45:04.168305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.168212    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"
Sep 17 15:45:04.790380 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.790358    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:45:04.840936 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.840872    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.840936 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.840905    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.840952    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841006    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841021    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841037    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841114    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wl7z\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841212    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs\") pod \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\" (UID: \"b3e7db46-3b2c-4954-a97b-29e9d7b51192\") "
Sep 17 15:45:04.841325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841260    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:04.841520 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841333    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data" (OuterVolumeSpecName: "istio-data") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:04.841520 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841390    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:04.841520 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841480    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:04.841716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841586    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:04.841935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841896    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.841935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841918    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.841935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841931    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.842133 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841940    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.842133 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.841949    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.843363 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.843332    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z" (OuterVolumeSpecName: "kube-api-access-5wl7z") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "kube-api-access-5wl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:04.843690 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.843669    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:45:04.843749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.843719    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:04.843749 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.843732    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token" (OuterVolumeSpecName: "istio-token") pod "b3e7db46-3b2c-4954-a97b-29e9d7b51192" (UID: "b3e7db46-3b2c-4954-a97b-29e9d7b51192"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:04.943449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.943424    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wl7z\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-kube-api-access-5wl7z\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.943449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.943449    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.943588 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.943459    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:04.943588 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:04.943467    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3e7db46-3b2c-4954-a97b-29e9d7b51192-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:05.177155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.177072    2564 generic.go:356] "Generic (PLEG): container finished" podID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerID="3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c" exitCode=0
Sep 17 15:45:05.177155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.177126    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"
Sep 17 15:45:05.177630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.177153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerDied","Data":"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"}
Sep 17 15:45:05.177630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.177216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt" event={"ID":"b3e7db46-3b2c-4954-a97b-29e9d7b51192","Type":"ContainerDied","Data":"31e9c528ce9aabc85613b5a0c291770c4e095d5ce6f794c66cd57180731218bb"}
Sep 17 15:45:05.177630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.177237    2564 scope.go:117] "RemoveContainer" containerID="3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"
Sep 17 15:45:05.186549 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.186530    2564 scope.go:117] "RemoveContainer" containerID="fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"
Sep 17 15:45:05.195159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.195143    2564 scope.go:117] "RemoveContainer" containerID="3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"
Sep 17 15:45:05.195411 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:05.195391    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c\": container with ID starting with 3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c not found: ID does not exist" containerID="3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"
Sep 17 15:45:05.195460 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.195422    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c"} err="failed to get container status \"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c\": rpc error: code = NotFound desc = could not find container \"3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c\": container with ID starting with 3394f18611bb8e709dca4e88d139d7ad8805006587b752e9a8b0085f2b151b1c not found: ID does not exist"
Sep 17 15:45:05.195460 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.195443    2564 scope.go:117] "RemoveContainer" containerID="fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"
Sep 17 15:45:05.195715 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:05.195699    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d\": container with ID starting with fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d not found: ID does not exist" containerID="fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"
Sep 17 15:45:05.195756 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.195721    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d"} err="failed to get container status \"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d\": rpc error: code = NotFound desc = could not find container \"fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d\": container with ID starting with fc18b23268855182e75e2f53aa7af1b0b0d63c43ba163193093bcec06f30c61d not found: ID does not exist"
Sep 17 15:45:05.202293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.202271    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:45:05.208729 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:05.208709    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6b2abe85-openshift-default-5f8fc7b478-4snzt"]
Sep 17 15:45:06.525707 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:06.525677    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" path="/var/lib/kubelet/pods/b3e7db46-3b2c-4954-a97b-29e9d7b51192/volumes"
Sep 17 15:45:07.276604 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:07.276575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:45:08.321101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321071    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:08.321480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321469    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.321523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321482    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.321523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321494    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed50bed4-416a-4012-86a5-8f072c0c4b4d" containerName="cleanup"
Sep 17 15:45:08.321523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321500    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed50bed4-416a-4012-86a5-8f072c0c4b4d" containerName="cleanup"
Sep 17 15:45:08.321523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321513    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f887bf0-c4df-4579-9126-7b87808584fe" containerName="cleanup"
Sep 17 15:45:08.321523 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321518    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f887bf0-c4df-4579-9126-7b87808584fe" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321535    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161e5807-e23c-4597-80d5-67a1363f1186" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321540    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="161e5807-e23c-4597-80d5-67a1363f1186" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321551    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f545ab57-26f0-4b1c-8061-2c4e4a39fde2" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321556    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f545ab57-26f0-4b1c-8061-2c4e4a39fde2" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321623    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="161e5807-e23c-4597-80d5-67a1363f1186" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321631    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f545ab57-26f0-4b1c-8061-2c4e4a39fde2" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321638    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321646    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f887bf0-c4df-4579-9126-7b87808584fe" containerName="cleanup"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321651    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.321724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.321657    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed50bed4-416a-4012-86a5-8f072c0c4b4d" containerName="cleanup"
Sep 17 15:45:08.326541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.326524    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.328855 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.328836    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-23da8b90-dockercfg-2bxt4\""
Sep 17 15:45:08.328855 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.328845    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-23da8b90\""
Sep 17 15:45:08.336261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.336242    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:08.380380 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.379740    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:08.380767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.380583    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.380767 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.380606    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3e7db46-3b2c-4954-a97b-29e9d7b51192" containerName="istio-proxy"
Sep 17 15:45:08.384703 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.384679    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.409637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.409293    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:08.478931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.478898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4ph7\" (UniqueName: \"kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.479091 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.478947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw8vn\" (UniqueName: \"kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.479143 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.479077    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.479143 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.479123    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.580524 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.580449    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4ph7\" (UniqueName: \"kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.580686 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.580607    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw8vn\" (UniqueName: \"kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.580763 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.580731    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.580830 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.580781    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.585993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.585975    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-23da8b90\""
Sep 17 15:45:08.592354 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.592336    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.592428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.592339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.598932 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.598911    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw8vn\" (UniqueName: \"kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn\") pod \"payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.599088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.599067    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4ph7\" (UniqueName: \"kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7\") pod \"payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") " pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.640934 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.640915    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-23da8b90-dockercfg-2bxt4\""
Sep 17 15:45:08.648729 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.648712    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:08.704706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.704641    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:08.929030 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.929004    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:08.929609 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:08.929585    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod929dd698_52e5_4426_ad81_07a889edc077.slice/crio-b271acaf4f0b101bc861b50dd75a8dd80a3c48a62f5e3ff9674c50efa28e928c WatchSource:0}: Error finding container b271acaf4f0b101bc861b50dd75a8dd80a3c48a62f5e3ff9674c50efa28e928c: Status 404 returned error can't find the container with id b271acaf4f0b101bc861b50dd75a8dd80a3c48a62f5e3ff9674c50efa28e928c
Sep 17 15:45:08.989661 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:08.989597    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cad8cad_67e8_4eed_8698_daf9e27c91d2.slice/crio-a2400a9ed2c4882673969247cf95766bb81a897bb7b366c0aba98b6483b1f494 WatchSource:0}: Error finding container a2400a9ed2c4882673969247cf95766bb81a897bb7b366c0aba98b6483b1f494: Status 404 returned error can't find the container with id a2400a9ed2c4882673969247cf95766bb81a897bb7b366c0aba98b6483b1f494
Sep 17 15:45:08.990163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:08.990137    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:09.207167 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.207077    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" event={"ID":"9cad8cad-67e8-4eed-8698-daf9e27c91d2","Type":"ContainerStarted","Data":"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"}
Sep 17 15:45:09.207330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.207168    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" event={"ID":"9cad8cad-67e8-4eed-8698-daf9e27c91d2","Type":"ContainerStarted","Data":"a2400a9ed2c4882673969247cf95766bb81a897bb7b366c0aba98b6483b1f494"}
Sep 17 15:45:09.207395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.207349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:09.209070 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.209044    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" event={"ID":"929dd698-52e5-4426-ad81-07a889edc077","Type":"ContainerStarted","Data":"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"}
Sep 17 15:45:09.209218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.209085    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" event={"ID":"929dd698-52e5-4426-ad81-07a889edc077","Type":"ContainerStarted","Data":"b271acaf4f0b101bc861b50dd75a8dd80a3c48a62f5e3ff9674c50efa28e928c"}
Sep 17 15:45:09.209218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.209182    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:09.235255 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.235213    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" podStartSLOduration=1.235200944 podStartE2EDuration="1.235200944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:09.232682078 +0000 UTC m=+901.199859895" watchObservedRunningTime="2026-09-17 15:45:09.235200944 +0000 UTC m=+901.202378754"
Sep 17 15:45:09.256218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:09.256181    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" podStartSLOduration=1.256168008 podStartE2EDuration="1.256168008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:09.25511372 +0000 UTC m=+901.222291540" watchObservedRunningTime="2026-09-17 15:45:09.256168008 +0000 UTC m=+901.223345825"
Sep 17 15:45:20.217514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:20.217474    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:20.218722 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:20.218630    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:25.308213 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.308182    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f962857-8ae0-481a-80fd-95755547031b" containerID="9debd76761f50fd90e1d2de7975a51da69d21caf458ccb90da0af41925bb8b1e" exitCode=0
Sep 17 15:45:25.308681 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.308252    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerDied","Data":"9debd76761f50fd90e1d2de7975a51da69d21caf458ccb90da0af41925bb8b1e"}
Sep 17 15:45:25.308760 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.308728    2564 scope.go:117] "RemoveContainer" containerID="9debd76761f50fd90e1d2de7975a51da69d21caf458ccb90da0af41925bb8b1e"
Sep 17 15:45:25.309995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.309966    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:25.310100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.310048    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:25.310162 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:25.310098    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:26.318435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:26.318397    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerStarted","Data":"13afac6f48bf0a23ff123c2a2dfd572002924a499dbaab73609e4bf881752ab4"}
Sep 17 15:45:26.664158 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:26.664075    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:26.664374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:26.664348    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:26.664458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:26.664406    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:27.664181 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:27.664143    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:27.664560 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:27.664201    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:27.720434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:27.720395    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:45:28.664072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:28.664031    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:28.664227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:28.664086    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:29.663911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:29.663877    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:29.664098 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:29.663935    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:30.663627 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:30.663590    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:30.664017 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:30.663672    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:31.664052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:31.664011    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 17 15:45:31.664431 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:31.664073    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 17 15:45:32.548156 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.548123    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"]
Sep 17 15:45:32.550912 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.550894    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:32.553140 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.553120    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:45:32.553250 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.553121    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:45:32.560968 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.560935    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"]
Sep 17 15:45:32.642815 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.642776    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv\" (UID: \"43a812bd-d4c4-4db8-8336-a2d741251b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:32.667418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.667386    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:32.667785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.667663    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:32.668377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.668359    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:32.744059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.744031    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv\" (UID: \"43a812bd-d4c4-4db8-8336-a2d741251b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:32.744732 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.744710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv\" (UID: \"43a812bd-d4c4-4db8-8336-a2d741251b4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:32.862724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:32.862642    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:33.135664 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:33.135640    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"]
Sep 17 15:45:33.137142 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:33.137113    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a812bd_d4c4_4db8_8336_a2d741251b4c.slice/crio-9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3 WatchSource:0}: Error finding container 9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3: Status 404 returned error can't find the container with id 9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3
Sep 17 15:45:33.367173 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:33.367144    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv" event={"ID":"43a812bd-d4c4-4db8-8336-a2d741251b4c","Type":"ContainerStarted","Data":"9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3"}
Sep 17 15:45:33.889155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:33.889123    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:45:34.380395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.380358    2564 generic.go:356] "Generic (PLEG): container finished" podID="43a812bd-d4c4-4db8-8336-a2d741251b4c" containerID="c9c11e2fd3d48d207891bea9fa1dff0de9b0afd4207ede87ccfd9d48a8ca1da1" exitCode=0
Sep 17 15:45:34.380593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.380441    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv" event={"ID":"43a812bd-d4c4-4db8-8336-a2d741251b4c","Type":"ContainerDied","Data":"c9c11e2fd3d48d207891bea9fa1dff0de9b0afd4207ede87ccfd9d48a8ca1da1"}
Sep 17 15:45:34.380759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.380734    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" containerID="cri-o://4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268" gracePeriod=30
Sep 17 15:45:34.664884 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.664805    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:45:34.667755 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.667733    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.670075 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.670055    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-5b08fa-openshift-default-dockercfg-574q2\""
Sep 17 15:45:34.680991 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.680967    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:45:34.775271 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.775229    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.775718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.775691    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.775909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.775894    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.776042 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.776027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.776174 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.776161    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.776316 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.776301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vdrk\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.777231 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.777204    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.777473 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.777454    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.777787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.777758    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879409 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vdrk\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879437    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879480    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879561    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879647    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879673    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879703    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.879856 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879735    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.880099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879863    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.880099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.879984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.880099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.880091    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.880220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.880142    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.880548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.880521    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.882035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.882008    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.882571 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.882552    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.891770 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.891745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.892214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.892192    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vdrk\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk\") pod \"e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") " pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:34.979792 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:34.979705    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:35.260684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.260652    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:45:35.264628 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:35.264584    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e04f0b2_23b9_4c5e_9d1a_332e9d6f83bb.slice/crio-b10955d274561facfc82d42ad85c798ef5a7b2d426d50c5f819dc7764565544b WatchSource:0}: Error finding container b10955d274561facfc82d42ad85c798ef5a7b2d426d50c5f819dc7764565544b: Status 404 returned error can't find the container with id b10955d274561facfc82d42ad85c798ef5a7b2d426d50c5f819dc7764565544b
Sep 17 15:45:35.266868 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.266835    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:35.266962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.266896    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:35.266962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.266926    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:45:35.392837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.392796    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerStarted","Data":"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"}
Sep 17 15:45:35.393025 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.392860    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerStarted","Data":"b10955d274561facfc82d42ad85c798ef5a7b2d426d50c5f819dc7764565544b"}
Sep 17 15:45:35.417278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.417214    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podStartSLOduration=1.417193557 podStartE2EDuration="1.417193557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:35.416165459 +0000 UTC m=+927.383343289" watchObservedRunningTime="2026-09-17 15:45:35.417193557 +0000 UTC m=+927.384371376"
Sep 17 15:45:35.573381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.573358    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:35.691626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.691580    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca\") pod \"43a812bd-d4c4-4db8-8336-a2d741251b4c\" (UID: \"43a812bd-d4c4-4db8-8336-a2d741251b4c\") "
Sep 17 15:45:35.691939 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.691914    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43a812bd-d4c4-4db8-8336-a2d741251b4c" (UID: "43a812bd-d4c4-4db8-8336-a2d741251b4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:35.792662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.792598    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a812bd-d4c4-4db8-8336-a2d741251b4c-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:35.980044 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:35.980011    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:36.400444 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:36.400419    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"
Sep 17 15:45:36.400639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:36.400469    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv" event={"ID":"43a812bd-d4c4-4db8-8336-a2d741251b4c","Type":"ContainerDied","Data":"9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3"}
Sep 17 15:45:36.400639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:36.400521    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9221cf96ec2b2b3023ba9972feea536ce4a272dbe2f868afbeaa16aed972ffc3"
Sep 17 15:45:36.985363 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:36.985333    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:37.407959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:37.407920    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:37.408900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:37.408877    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:45:39.087362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.087320    2564 patch_prober.go:28] interesting pod/e2e-autores-28eba6-openshift-default-544d775478-s6m9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:45:39.087845 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.087382    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:45:39.609338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.609300    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:45:39.609995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.609978    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a812bd-d4c4-4db8-8336-a2d741251b4c" containerName="revoke-keys"
Sep 17 15:45:39.609995 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.609997    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a812bd-d4c4-4db8-8336-a2d741251b4c" containerName="revoke-keys"
Sep 17 15:45:39.610126 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.610114    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a812bd-d4c4-4db8-8336-a2d741251b4c" containerName="revoke-keys"
Sep 17 15:45:39.613942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.613923    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.616915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.616893    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-5b08fa\""
Sep 17 15:45:39.616993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.616949    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-5b08fa-dockercfg-zwmzx\""
Sep 17 15:45:39.628147 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.628120    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:45:39.637799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.637772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:45:39.737685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737052    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737646    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737763    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph96w\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737804    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737818    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737871    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.737940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.738263 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.737998    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:39.738263 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738008    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.738263 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket\") pod \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\" (UID: \"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7\") "
Sep 17 15:45:39.738263 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738133    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738787    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wz7b\" (UniqueName: \"kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738924    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738939    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738955    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.739015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.738962    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:39.739442 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.739164    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:39.743210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.741816    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:39.743542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.743421    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w" (OuterVolumeSpecName: "kube-api-access-ph96w") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "kube-api-access-ph96w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:39.743542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.743501    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:39.743723 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.743626    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" (UID: "7aaa06c9-115f-4c2a-9f96-3dc7c95842f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:45:39.840013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.839978    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.840013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840022    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wz7b\" (UniqueName: \"kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840099    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840108    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840117    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840125    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840134    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph96w\" (UniqueName: \"kubernetes.io/projected/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-kube-api-access-ph96w\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840142    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:39.840771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.840752    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.856987 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.856960    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wz7b\" (UniqueName: \"kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b\") pod \"payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") " pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:39.944711 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:39.944669    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:40.244064 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.244034    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:45:40.244987 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:45:40.244962    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8efd57_51bf_4c1b_8495_0b767f8b1e84.slice/crio-40298c4e77ae6cc1fecb0c81fa2c0b4532f78831771acabcf065acdc4f06e604 WatchSource:0}: Error finding container 40298c4e77ae6cc1fecb0c81fa2c0b4532f78831771acabcf065acdc4f06e604: Status 404 returned error can't find the container with id 40298c4e77ae6cc1fecb0c81fa2c0b4532f78831771acabcf065acdc4f06e604
Sep 17 15:45:40.426439 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.426408    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" event={"ID":"ed8efd57-51bf-4c1b-8495-0b767f8b1e84","Type":"ContainerStarted","Data":"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"}
Sep 17 15:45:40.426639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.426471    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" event={"ID":"ed8efd57-51bf-4c1b-8495-0b767f8b1e84","Type":"ContainerStarted","Data":"40298c4e77ae6cc1fecb0c81fa2c0b4532f78831771acabcf065acdc4f06e604"}
Sep 17 15:45:40.426639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.426501    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:40.428181 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.428157    2564 generic.go:356] "Generic (PLEG): container finished" podID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerID="4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268" exitCode=0
Sep 17 15:45:40.428283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.428196    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerDied","Data":"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"}
Sep 17 15:45:40.428283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.428202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"
Sep 17 15:45:40.428283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.428242    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w" event={"ID":"7aaa06c9-115f-4c2a-9f96-3dc7c95842f7","Type":"ContainerDied","Data":"11aef9cf9dcd661e8ff996e813f5763ca61900e13705668b7885d7d0c0211c12"}
Sep 17 15:45:40.428283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.428274    2564 scope.go:117] "RemoveContainer" containerID="4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"
Sep 17 15:45:40.438558 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.438540    2564 scope.go:117] "RemoveContainer" containerID="46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"
Sep 17 15:45:40.446646 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.446626    2564 scope.go:117] "RemoveContainer" containerID="4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"
Sep 17 15:45:40.446936 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:40.446915    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268\": container with ID starting with 4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268 not found: ID does not exist" containerID="4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"
Sep 17 15:45:40.447035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.446947    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268"} err="failed to get container status \"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268\": rpc error: code = NotFound desc = could not find container \"4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268\": container with ID starting with 4f7d96f815f82e00eb55c9944eb4b92ce2043a7050a4c0952c94c8758ad60268 not found: ID does not exist"
Sep 17 15:45:40.447035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.446969    2564 scope.go:117] "RemoveContainer" containerID="46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"
Sep 17 15:45:40.447234 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:40.447214    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c\": container with ID starting with 46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c not found: ID does not exist" containerID="46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"
Sep 17 15:45:40.447276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.447243    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c"} err="failed to get container status \"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c\": rpc error: code = NotFound desc = could not find container \"46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c\": container with ID starting with 46b510c6495af522d33d083f3c56f51dc0115103a526f886962b121990e4395c not found: ID does not exist"
Sep 17 15:45:40.457939 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.457889    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" podStartSLOduration=1.457872706 podStartE2EDuration="1.457872706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:45:40.456004097 +0000 UTC m=+932.423181916" watchObservedRunningTime="2026-09-17 15:45:40.457872706 +0000 UTC m=+932.425050526"
Sep 17 15:45:40.475366 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.475331    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:45:40.478009 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.477989    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-28eba6-openshift-default-544d775478-s6m9w"]
Sep 17 15:45:40.535159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:40.528798    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" path="/var/lib/kubelet/pods/7aaa06c9-115f-4c2a-9f96-3dc7c95842f7/volumes"
Sep 17 15:45:44.932700 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:44.932664    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:44.933139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:44.932883    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" podUID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" containerName="payload-processing" containerID="cri-o://b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695" gracePeriod=30
Sep 17 15:45:44.941136 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:44.941112    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:44.941352 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:44.941323    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" podUID="929dd698-52e5-4426-ad81-07a889edc077" containerName="payload-pre-processing" containerID="cri-o://177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756" gracePeriod=30
Sep 17 15:45:45.154413 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.154382    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"]
Sep 17 15:45:45.160048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.160010    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-23da8b90-zfjqv"]
Sep 17 15:45:45.191828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.191768    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:45.231753 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.231726    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:45.299958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.299120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4ph7\" (UniqueName: \"kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7\") pod \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") "
Sep 17 15:45:45.299958 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.299287    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume\") pod \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\" (UID: \"9cad8cad-67e8-4eed-8698-daf9e27c91d2\") "
Sep 17 15:45:45.300548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.300401    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9cad8cad-67e8-4eed-8698-daf9e27c91d2" (UID: "9cad8cad-67e8-4eed-8698-daf9e27c91d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:45.304233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.304121    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7" (OuterVolumeSpecName: "kube-api-access-p4ph7") pod "9cad8cad-67e8-4eed-8698-daf9e27c91d2" (UID: "9cad8cad-67e8-4eed-8698-daf9e27c91d2"). InnerVolumeSpecName "kube-api-access-p4ph7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:45.307159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.307124    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:45.308136 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.308108    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" containerID="cri-o://13afac6f48bf0a23ff123c2a2dfd572002924a499dbaab73609e4bf881752ab4" gracePeriod=30
Sep 17 15:45:45.400460 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.400423    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume\") pod \"929dd698-52e5-4426-ad81-07a889edc077\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") "
Sep 17 15:45:45.400817 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.400795    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw8vn\" (UniqueName: \"kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn\") pod \"929dd698-52e5-4426-ad81-07a889edc077\" (UID: \"929dd698-52e5-4426-ad81-07a889edc077\") "
Sep 17 15:45:45.401370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.401049    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "929dd698-52e5-4426-ad81-07a889edc077" (UID: "929dd698-52e5-4426-ad81-07a889edc077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:45.401370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.401308    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cad8cad-67e8-4eed-8698-daf9e27c91d2-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:45.401370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.401328    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/929dd698-52e5-4426-ad81-07a889edc077-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:45.401370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.401344    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4ph7\" (UniqueName: \"kubernetes.io/projected/9cad8cad-67e8-4eed-8698-daf9e27c91d2-kube-api-access-p4ph7\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:45.414546 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.414515    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn" (OuterVolumeSpecName: "kube-api-access-tw8vn") pod "929dd698-52e5-4426-ad81-07a889edc077" (UID: "929dd698-52e5-4426-ad81-07a889edc077"). InnerVolumeSpecName "kube-api-access-tw8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:45.461254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.461158    2564 generic.go:356] "Generic (PLEG): container finished" podID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" containerID="b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695" exitCode=0
Sep 17 15:45:45.461254 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.461204    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"
Sep 17 15:45:45.461474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.461247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" event={"ID":"9cad8cad-67e8-4eed-8698-daf9e27c91d2","Type":"ContainerDied","Data":"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"}
Sep 17 15:45:45.461474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.461300    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt" event={"ID":"9cad8cad-67e8-4eed-8698-daf9e27c91d2","Type":"ContainerDied","Data":"a2400a9ed2c4882673969247cf95766bb81a897bb7b366c0aba98b6483b1f494"}
Sep 17 15:45:45.461474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.461328    2564 scope.go:117] "RemoveContainer" containerID="b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"
Sep 17 15:45:45.462880 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.462856    2564 generic.go:356] "Generic (PLEG): container finished" podID="929dd698-52e5-4426-ad81-07a889edc077" containerID="177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756" exitCode=0
Sep 17 15:45:45.462982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.462906    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"
Sep 17 15:45:45.462982 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.462955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" event={"ID":"929dd698-52e5-4426-ad81-07a889edc077","Type":"ContainerDied","Data":"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"}
Sep 17 15:45:45.463059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.463000    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d" event={"ID":"929dd698-52e5-4426-ad81-07a889edc077","Type":"ContainerDied","Data":"b271acaf4f0b101bc861b50dd75a8dd80a3c48a62f5e3ff9674c50efa28e928c"}
Sep 17 15:45:45.478488 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.478470    2564 scope.go:117] "RemoveContainer" containerID="b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"
Sep 17 15:45:45.478862 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:45.478843    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695\": container with ID starting with b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695 not found: ID does not exist" containerID="b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"
Sep 17 15:45:45.478955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.478874    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695"} err="failed to get container status \"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695\": rpc error: code = NotFound desc = could not find container \"b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695\": container with ID starting with b1f1ad947a581cc0a2c49583dcf98b7612ae807ea4fa470cbdbcd909e2e8c695 not found: ID does not exist"
Sep 17 15:45:45.478955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.478897    2564 scope.go:117] "RemoveContainer" containerID="177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"
Sep 17 15:45:45.487251 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.487230    2564 scope.go:117] "RemoveContainer" containerID="177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"
Sep 17 15:45:45.487496 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:45:45.487477    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756\": container with ID starting with 177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756 not found: ID does not exist" containerID="177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"
Sep 17 15:45:45.487535 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.487505    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756"} err="failed to get container status \"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756\": rpc error: code = NotFound desc = could not find container \"177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756\": container with ID starting with 177b274a6cfaa80583806beba92ab3c58c89ee4b162f2e782046c4e6c9569756 not found: ID does not exist"
Sep 17 15:45:45.494367 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.494329    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:45.496971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.496949    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-23da8b90-54b7b6d9fd-p7f2d"]
Sep 17 15:45:45.502830 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.502807    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw8vn\" (UniqueName: \"kubernetes.io/projected/929dd698-52e5-4426-ad81-07a889edc077-kube-api-access-tw8vn\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:45.511968 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.511942    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:45.514345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:45.514323    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-23da8b90-7b9cb4bd77-gp6mt"]
Sep 17 15:45:46.525787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:46.525756    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43a812bd-d4c4-4db8-8336-a2d741251b4c" path="/var/lib/kubelet/pods/43a812bd-d4c4-4db8-8336-a2d741251b4c/volumes"
Sep 17 15:45:46.526160 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:46.526122    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="929dd698-52e5-4426-ad81-07a889edc077" path="/var/lib/kubelet/pods/929dd698-52e5-4426-ad81-07a889edc077/volumes"
Sep 17 15:45:46.526432 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:46.526420    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" path="/var/lib/kubelet/pods/9cad8cad-67e8-4eed-8698-daf9e27c91d2/volumes"
Sep 17 15:45:47.670762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:47.670725    2564 patch_prober.go:28] interesting pod/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:45:47.671213 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:47.670793    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:45:50.499082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.499045    2564 generic.go:356] "Generic (PLEG): container finished" podID="1f962857-8ae0-481a-80fd-95755547031b" containerID="13afac6f48bf0a23ff123c2a2dfd572002924a499dbaab73609e4bf881752ab4" exitCode=0
Sep 17 15:45:50.499514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.499111    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerDied","Data":"13afac6f48bf0a23ff123c2a2dfd572002924a499dbaab73609e4bf881752ab4"}
Sep 17 15:45:50.499514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.499209    2564 scope.go:117] "RemoveContainer" containerID="9debd76761f50fd90e1d2de7975a51da69d21caf458ccb90da0af41925bb8b1e"
Sep 17 15:45:50.559911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.559889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:50.659370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659295    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659367    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659396    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg2gq\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659428    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659491    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659528    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659563    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659626    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659663    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket\") pod \"1f962857-8ae0-481a-80fd-95755547031b\" (UID: \"1f962857-8ae0-481a-80fd-95755547031b\") "
Sep 17 15:45:50.659764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659723    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:50.660039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659794    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:50.660039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659884    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:45:50.660039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.659934    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data" (OuterVolumeSpecName: "istio-data") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:50.660204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.660086    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:50.660204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.660122    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.660204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.660143    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f962857-8ae0-481a-80fd-95755547031b-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.660204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.660156    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.660204 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.660168    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.661677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.661648    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:45:50.661803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.661736    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq" (OuterVolumeSpecName: "kube-api-access-lg2gq") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "kube-api-access-lg2gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:50.661803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.661772    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:45:50.661972 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.661953    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1f962857-8ae0-481a-80fd-95755547031b" (UID: "1f962857-8ae0-481a-80fd-95755547031b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:45:50.761329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.761303    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.761329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.761326    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.761482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.761335    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f962857-8ae0-481a-80fd-95755547031b-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.761482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.761343    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f962857-8ae0-481a-80fd-95755547031b-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:50.761482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:50.761352    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg2gq\" (UniqueName: \"kubernetes.io/projected/1f962857-8ae0-481a-80fd-95755547031b-kube-api-access-lg2gq\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:45:51.438562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.438534    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:45:51.508173 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.508146    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"
Sep 17 15:45:51.508604 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.508192    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk" event={"ID":"1f962857-8ae0-481a-80fd-95755547031b","Type":"ContainerDied","Data":"7e9943f1697a7455d2f0aedce746cb677a9ce4c38a2ba6d857e072545b52599c"}
Sep 17 15:45:51.508604 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.508256    2564 scope.go:117] "RemoveContainer" containerID="13afac6f48bf0a23ff123c2a2dfd572002924a499dbaab73609e4bf881752ab4"
Sep 17 15:45:51.535235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.535211    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:51.539287 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:51.539263    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-23da8b90-openshift-default-5c55dcc496-8rkfk"]
Sep 17 15:45:52.525423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:52.525393    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f962857-8ae0-481a-80fd-95755547031b" path="/var/lib/kubelet/pods/1f962857-8ae0-481a-80fd-95755547031b/volumes"
Sep 17 15:45:54.684499 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:45:54.684466    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:46:02.078490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078451    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078892    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078904    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078919    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078925    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078937    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:46:02.078951 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078944    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078961    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" containerName="payload-processing"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078967    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" containerName="payload-processing"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078976    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078981    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078988    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="929dd698-52e5-4426-ad81-07a889edc077" containerName="payload-pre-processing"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.078993    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="929dd698-52e5-4426-ad81-07a889edc077" containerName="payload-pre-processing"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.079047    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.079055    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.079061    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cad8cad-67e8-4eed-8698-daf9e27c91d2" containerName="payload-processing"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.079068    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aaa06c9-115f-4c2a-9f96-3dc7c95842f7" containerName="istio-proxy"
Sep 17 15:46:02.079266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.079074    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="929dd698-52e5-4426-ad81-07a889edc077" containerName="payload-pre-processing"
Sep 17 15:46:02.083830 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.083810    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.096101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.096063    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a2dc23\"/\"kube-root-ca.crt\""
Sep 17 15:46:02.096215 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.096101    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a2dc23\"/\"default-dockercfg-j4d5c\""
Sep 17 15:46:02.096215 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.096065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a2dc23\"/\"test-model-a2dc23-kserve-self-signed-certs\""
Sep 17 15:46:02.096338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.096262    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a2dc23\"/\"openshift-service-ca.crt\""
Sep 17 15:46:02.102915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.102893    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:46:02.177574 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177535    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.177745 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177577    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.177745 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnrh\" (UniqueName: \"kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.177906 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177752    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.177906 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177833    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.177985 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.177920    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.278849 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.278815    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.278849 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.278853    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.278909    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnrh\" (UniqueName: \"kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.278934    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.279093    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.279195    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.279256    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279589 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.279561    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.279694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.279594    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.281212 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.281182    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.281582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.281568    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.289522 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.289494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnrh\" (UniqueName: \"kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh\") pod \"test-model-a2dc23-kserve-75b5b8bf55-44m2p\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") " pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.393446 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.393353    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:02.690947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:02.690920    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:46:02.691984 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:46:02.691957    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab21152d_21df_4d71_bd8b_0da3bc0bc354.slice/crio-d3d576f3afa5b9faa98189ec13e1861ca548dbbe5dc65e2025b5fea177067772 WatchSource:0}: Error finding container d3d576f3afa5b9faa98189ec13e1861ca548dbbe5dc65e2025b5fea177067772: Status 404 returned error can't find the container with id d3d576f3afa5b9faa98189ec13e1861ca548dbbe5dc65e2025b5fea177067772
Sep 17 15:46:03.586457 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:03.586414    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" event={"ID":"ab21152d-21df-4d71-bd8b-0da3bc0bc354","Type":"ContainerStarted","Data":"5f4b47fb209981f24db9fd897c121b8a727648bc0ae562a7d043585c3913330f"}
Sep 17 15:46:03.586854 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:03.586474    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" event={"ID":"ab21152d-21df-4d71-bd8b-0da3bc0bc354","Type":"ContainerStarted","Data":"d3d576f3afa5b9faa98189ec13e1861ca548dbbe5dc65e2025b5fea177067772"}
Sep 17 15:46:03.607903 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:03.607839    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" podStartSLOduration=1.607819009 podStartE2EDuration="1.607819009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:46:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:46:03.605208711 +0000 UTC m=+955.572386544" watchObservedRunningTime="2026-09-17 15:46:03.607819009 +0000 UTC m=+955.574996806"
Sep 17 15:46:05.600148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.600110    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerID="df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f" exitCode=0
Sep 17 15:46:05.600577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.600191    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerDied","Data":"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"}
Sep 17 15:46:05.600723 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.600705    2564 scope.go:117] "RemoveContainer" containerID="df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"
Sep 17 15:46:05.602139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.602110    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:46:05.602218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.602182    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:46:05.602218 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:05.602214    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:46:06.610930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:06.610884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerStarted","Data":"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"}
Sep 17 15:46:06.980219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:06.980130    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:46:06.981535 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:06.981506    2564 patch_prober.go:28] interesting pod/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 15:46:06.981672 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:06.981578    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 15:46:07.408952 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:07.408910    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:46:07.980691 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:07.980648    2564 patch_prober.go:28] interesting pod/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 15:46:07.981073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:07.980707    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 15:46:08.980910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:08.980823    2564 patch_prober.go:28] interesting pod/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 15:46:08.980910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:08.980888    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 15:46:09.980427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:09.980395    2564 patch_prober.go:28] interesting pod/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 15:46:09.980637 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:09.980463    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 15:46:10.981183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:10.981145    2564 patch_prober.go:28] interesting pod/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body=
Sep 17 15:46:10.981562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:10.981201    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused"
Sep 17 15:46:11.984437 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:11.984408    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:46:11.985434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:11.985416    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:46:12.394124 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:12.394088    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:12.394300 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:12.394139    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:12.407259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:12.407229    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:12.665765 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:12.665693    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:46:15.679657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:15.679599    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:46:40.183776 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:40.183740    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:46:58.998600 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:46:58.998565    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:47:00.086842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.086804    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:00.087485 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.087463    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f962857-8ae0-481a-80fd-95755547031b" containerName="istio-proxy"
Sep 17 15:47:00.089758 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.089738    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.093400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.093379    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5b08fa\"/\"kube-root-ca.crt\""
Sep 17 15:47:00.093501 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.093404    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-5b08fa\"/\"openshift-service-ca.crt\""
Sep 17 15:47:00.094103 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.094086    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5b08fa\"/\"explicit-model-5b08fa-kserve-self-signed-certs\""
Sep 17 15:47:00.096169 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.096153    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-5b08fa\"/\"default-dockercfg-prhlf\""
Sep 17 15:47:00.122522 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.122497    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:00.180626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180576    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznch\" (UniqueName: \"kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.180774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180701    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.180774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.180892 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180774    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.180892 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.181001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.180906    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282169 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282140    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznch\" (UniqueName: \"kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282302    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282346    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282372    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282431    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282600 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282522    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282770    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.282955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282887    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.283023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.282970    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.285050 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.285024    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.285247 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.285228    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.295805 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.295755    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznch\" (UniqueName: \"kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch\") pod \"explicit-model-5b08fa-kserve-7cb5995469-l8s2r\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") " pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.399334 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.399256    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:00.682506 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.682478    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:00.684190 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:00.684168    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8f42d2_37cf_4680_8e35_79136a4b3be1.slice/crio-7762d274d07df9e61878ea01279cc6d8288d00d9cf4e1215f94885ab616c6a03 WatchSource:0}: Error finding container 7762d274d07df9e61878ea01279cc6d8288d00d9cf4e1215f94885ab616c6a03: Status 404 returned error can't find the container with id 7762d274d07df9e61878ea01279cc6d8288d00d9cf4e1215f94885ab616c6a03
Sep 17 15:47:00.942559 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.942473    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" event={"ID":"fd8f42d2-37cf-4680-8e35-79136a4b3be1","Type":"ContainerStarted","Data":"a1359d991094a1a6918ebbcfe3002686313efde1080c359ed09fec2bf0bf5b4e"}
Sep 17 15:47:00.942559 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.942525    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" event={"ID":"fd8f42d2-37cf-4680-8e35-79136a4b3be1","Type":"ContainerStarted","Data":"7762d274d07df9e61878ea01279cc6d8288d00d9cf4e1215f94885ab616c6a03"}
Sep 17 15:47:00.969304 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:00.969259    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" podStartSLOduration=0.969242573 podStartE2EDuration="969.242573ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:00.968703694 +0000 UTC m=+1012.935881515" watchObservedRunningTime="2026-09-17 15:47:00.969242573 +0000 UTC m=+1012.936420390"
Sep 17 15:47:05.652146 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.652113    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:47:05.655071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.655050    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.658103 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.658078    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"e2e-conflict-a-bd4c6f71-kserve-self-signed-certs\""
Sep 17 15:47:05.659067 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.659044    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"default-dockercfg-287z2\""
Sep 17 15:47:05.659067 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.659049    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"openshift-service-ca.crt\""
Sep 17 15:47:05.659241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.659105    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"kube-root-ca.crt\""
Sep 17 15:47:05.666141 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.666117    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:47:05.738799 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.738763    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.738975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.738822    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4sg\" (UniqueName: \"kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.738975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.738853    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.738975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.738898    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.738975 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.738930    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.739143 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.739009    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840303 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840268    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840366    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4sg\" (UniqueName: \"kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840540    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840603    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840783 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840663    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840783 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840745    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.840886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.840824    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.841072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.841042    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.842832 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.842813    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.843118 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.843099    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.850136 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.850114    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4sg\" (UniqueName: \"kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg\") pod \"e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:05.965880 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:05.965801    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:06.243992 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:06.243965    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:47:06.246000 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:06.245975    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod496cd31b_be97_4723_a260_886153fc3359.slice/crio-1bb2a97774b9edfb6f9b5df7d88b14f3e6ab628c7a0aae902df2b491a1487493 WatchSource:0}: Error finding container 1bb2a97774b9edfb6f9b5df7d88b14f3e6ab628c7a0aae902df2b491a1487493: Status 404 returned error can't find the container with id 1bb2a97774b9edfb6f9b5df7d88b14f3e6ab628c7a0aae902df2b491a1487493
Sep 17 15:47:06.977904 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:06.977868    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" event={"ID":"496cd31b-be97-4723-a260-886153fc3359","Type":"ContainerStarted","Data":"60fc4042f3ba0a71913dfce23850010edec356954ee41cc44c62f52f7e8be135"}
Sep 17 15:47:06.978301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:06.977914    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" event={"ID":"496cd31b-be97-4723-a260-886153fc3359","Type":"ContainerStarted","Data":"1bb2a97774b9edfb6f9b5df7d88b14f3e6ab628c7a0aae902df2b491a1487493"}
Sep 17 15:47:08.927404 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:08.927342    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" podStartSLOduration=3.927326463 podStartE2EDuration="3.927326463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:07.011675455 +0000 UTC m=+1018.978853275" watchObservedRunningTime="2026-09-17 15:47:08.927326463 +0000 UTC m=+1020.894504281"
Sep 17 15:47:08.929524 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:08.929503    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:09.057216 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.057186    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"]
Sep 17 15:47:09.060026 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.060009    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:09.062495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.062474    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:47:09.062598 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.062479    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:47:09.071527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.071506    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"]
Sep 17 15:47:09.174318 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.174280    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw\" (UID: \"2dbeeeed-ab71-46a5-a4db-730c3799e029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:09.275726 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.275695    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw\" (UID: \"2dbeeeed-ab71-46a5-a4db-730c3799e029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:09.276293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.276276    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw\" (UID: \"2dbeeeed-ab71-46a5-a4db-730c3799e029\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:09.370463 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.370435    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:09.651302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.651274    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"]
Sep 17 15:47:09.653639 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:09.653586    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbeeeed_ab71_46a5_a4db_730c3799e029.slice/crio-cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8 WatchSource:0}: Error finding container cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8: Status 404 returned error can't find the container with id cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8
Sep 17 15:47:09.997025 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.996937    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw" event={"ID":"2dbeeeed-ab71-46a5-a4db-730c3799e029","Type":"ContainerStarted","Data":"328eac68ca7810d0d15e2ac7a66a44e1c89a2ad19cb5febf13786ad1fa2c6f10"}
Sep 17 15:47:09.997378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:09.997032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw" event={"ID":"2dbeeeed-ab71-46a5-a4db-730c3799e029","Type":"ContainerStarted","Data":"cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8"}
Sep 17 15:47:10.018232 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:10.018176    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw" podStartSLOduration=1.018160963 podStartE2EDuration="1.018160963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:10.016949255 +0000 UTC m=+1021.984127076" watchObservedRunningTime="2026-09-17 15:47:10.018160963 +0000 UTC m=+1021.985338780"
Sep 17 15:47:10.400291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:10.400256    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:11.004911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:11.004874    2564 generic.go:356] "Generic (PLEG): container finished" podID="2dbeeeed-ab71-46a5-a4db-730c3799e029" containerID="328eac68ca7810d0d15e2ac7a66a44e1c89a2ad19cb5febf13786ad1fa2c6f10" exitCode=0
Sep 17 15:47:11.005383 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:11.004955    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw" event={"ID":"2dbeeeed-ab71-46a5-a4db-730c3799e029","Type":"ContainerDied","Data":"328eac68ca7810d0d15e2ac7a66a44e1c89a2ad19cb5febf13786ad1fa2c6f10"}
Sep 17 15:47:12.157321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:12.157299    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:12.312102 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:12.312030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca\") pod \"2dbeeeed-ab71-46a5-a4db-730c3799e029\" (UID: \"2dbeeeed-ab71-46a5-a4db-730c3799e029\") "
Sep 17 15:47:12.312307 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:12.312286    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2dbeeeed-ab71-46a5-a4db-730c3799e029" (UID: "2dbeeeed-ab71-46a5-a4db-730c3799e029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:47:12.413144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:12.413107    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dbeeeed-ab71-46a5-a4db-730c3799e029-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:13.019874 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:13.019840    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"
Sep 17 15:47:13.020056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:13.019875    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw" event={"ID":"2dbeeeed-ab71-46a5-a4db-730c3799e029","Type":"ContainerDied","Data":"cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8"}
Sep 17 15:47:13.020056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:13.019931    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd5e2212a6792fb5a0553ba55e863dd1e5a28fe2a0832a0a6c90efaa11a4a9a8"
Sep 17 15:47:15.966428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:15.966384    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:15.966428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:15.966439    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:15.978986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:15.978964    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:16.048926 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:16.048901    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:47:20.465060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.465015    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:47:20.465570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.465245    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" podUID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" containerName="payload-processing" containerID="cri-o://8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719" gracePeriod=30
Sep 17 15:47:20.715766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.715700    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:47:20.896993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.896956    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume\") pod \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") "
Sep 17 15:47:20.897196 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.897036    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wz7b\" (UniqueName: \"kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b\") pod \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\" (UID: \"ed8efd57-51bf-4c1b-8495-0b767f8b1e84\") "
Sep 17 15:47:20.897391 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.897366    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed8efd57-51bf-4c1b-8495-0b767f8b1e84" (UID: "ed8efd57-51bf-4c1b-8495-0b767f8b1e84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:47:20.897489 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.897475    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:20.899101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.899076    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b" (OuterVolumeSpecName: "kube-api-access-8wz7b") pod "ed8efd57-51bf-4c1b-8495-0b767f8b1e84" (UID: "ed8efd57-51bf-4c1b-8495-0b767f8b1e84"). InnerVolumeSpecName "kube-api-access-8wz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:20.998867 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:20.998792    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wz7b\" (UniqueName: \"kubernetes.io/projected/ed8efd57-51bf-4c1b-8495-0b767f8b1e84-kube-api-access-8wz7b\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:21.068521 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.068478    2564 generic.go:356] "Generic (PLEG): container finished" podID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" containerID="8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719" exitCode=0
Sep 17 15:47:21.068701 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.068554    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"
Sep 17 15:47:21.068701 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.068565    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" event={"ID":"ed8efd57-51bf-4c1b-8495-0b767f8b1e84","Type":"ContainerDied","Data":"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"}
Sep 17 15:47:21.068701 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.068684    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj" event={"ID":"ed8efd57-51bf-4c1b-8495-0b767f8b1e84","Type":"ContainerDied","Data":"40298c4e77ae6cc1fecb0c81fa2c0b4532f78831771acabcf065acdc4f06e604"}
Sep 17 15:47:21.068815 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.068711    2564 scope.go:117] "RemoveContainer" containerID="8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"
Sep 17 15:47:21.078164 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.078144    2564 scope.go:117] "RemoveContainer" containerID="8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"
Sep 17 15:47:21.078440 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:47:21.078420    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719\": container with ID starting with 8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719 not found: ID does not exist" containerID="8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"
Sep 17 15:47:21.078510 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.078450    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719"} err="failed to get container status \"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719\": rpc error: code = NotFound desc = could not find container \"8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719\": container with ID starting with 8acf1862b26a030b2193f19b053c2c9ff2260ae59e3bb566aceec64ab6c05719 not found: ID does not exist"
Sep 17 15:47:21.135606 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.135577    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:47:21.137810 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.137788    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-5b08fa-848bcd5f5c-454kj"]
Sep 17 15:47:21.242228 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.242198    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"]
Sep 17 15:47:21.247096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:21.247070    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-5b08fa-9xwlw"]
Sep 17 15:47:22.526148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:22.526118    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbeeeed-ab71-46a5-a4db-730c3799e029" path="/var/lib/kubelet/pods/2dbeeeed-ab71-46a5-a4db-730c3799e029/volumes"
Sep 17 15:47:22.526527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:22.526475    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" path="/var/lib/kubelet/pods/ed8efd57-51bf-4c1b-8495-0b767f8b1e84/volumes"
Sep 17 15:47:23.948586 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:23.948539    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" podUID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" containerName="main" containerID="cri-o://a1359d991094a1a6918ebbcfe3002686313efde1080c359ed09fec2bf0bf5b4e" gracePeriod=45
Sep 17 15:47:24.091001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.090494    2564 generic.go:356] "Generic (PLEG): container finished" podID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" containerID="a1359d991094a1a6918ebbcfe3002686313efde1080c359ed09fec2bf0bf5b4e" exitCode=0
Sep 17 15:47:24.091001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.090602    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" event={"ID":"fd8f42d2-37cf-4680-8e35-79136a4b3be1","Type":"ContainerDied","Data":"a1359d991094a1a6918ebbcfe3002686313efde1080c359ed09fec2bf0bf5b4e"}
Sep 17 15:47:24.196395 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.196370    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:24.232803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.232715    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.232960 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.232840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.232960 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.232879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.232960 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.232917    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.233129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.232997    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.233129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233062    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sznch\" (UniqueName: \"kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch\") pod \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\" (UID: \"fd8f42d2-37cf-4680-8e35-79136a4b3be1\") "
Sep 17 15:47:24.233305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233279    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:24.233428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233388    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache" (OuterVolumeSpecName: "model-cache") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:24.233531 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233504    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home" (OuterVolumeSpecName: "home") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:24.233800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233783    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.233897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233803    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.233897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.233817    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.235283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.235256    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:47:24.235375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.235334    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch" (OuterVolumeSpecName: "kube-api-access-sznch") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "kube-api-access-sznch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:24.235375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.235367    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm" (OuterVolumeSpecName: "dshm") pod "fd8f42d2-37cf-4680-8e35-79136a4b3be1" (UID: "fd8f42d2-37cf-4680-8e35-79136a4b3be1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:24.236567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.236545    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:47:24.237002 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.236988    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" containerName="payload-processing"
Sep 17 15:47:24.237052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237004    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" containerName="payload-processing"
Sep 17 15:47:24.237052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237019    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbeeeed-ab71-46a5-a4db-730c3799e029" containerName="revoke-keys"
Sep 17 15:47:24.237052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237025    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbeeeed-ab71-46a5-a4db-730c3799e029" containerName="revoke-keys"
Sep 17 15:47:24.237052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237040    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" containerName="main"
Sep 17 15:47:24.237052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237046    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" containerName="main"
Sep 17 15:47:24.237205 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237109    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" containerName="main"
Sep 17 15:47:24.237205 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237117    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbeeeed-ab71-46a5-a4db-730c3799e029" containerName="revoke-keys"
Sep 17 15:47:24.237205 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.237124    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed8efd57-51bf-4c1b-8495-0b767f8b1e84" containerName="payload-processing"
Sep 17 15:47:24.248424 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.248405    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.252656 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.252635    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"e2e-conflict-b-bd4c6f71-kserve-self-signed-certs\""
Sep 17 15:47:24.295802 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.295766    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:47:24.334474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334435    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334491    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334516    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334605    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zrtz\" (UniqueName: \"kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.334860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334827    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f42d2-37cf-4680-8e35-79136a4b3be1-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.334860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334845    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sznch\" (UniqueName: \"kubernetes.io/projected/fd8f42d2-37cf-4680-8e35-79136a4b3be1-kube-api-access-sznch\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.334999 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.334860    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f42d2-37cf-4680-8e35-79136a4b3be1-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:24.436177 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436131    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436177 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436175    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436306    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436336    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436408    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zrtz\" (UniqueName: \"kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436689    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436736    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.436766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.436748    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.438447 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.438412    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.438731 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.438710    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.452513 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.452491    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zrtz\" (UniqueName: \"kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz\") pod \"e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") " pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.558560 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.558536    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:24.844013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:24.843988    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:47:24.845363 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:24.845337    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfdc4b642_4115_4c21_85fb_2967bebdbd59.slice/crio-f13eb7e1f0250646b37fb480a00713221b5149dd411a7d0f329a4b28f71846f9 WatchSource:0}: Error finding container f13eb7e1f0250646b37fb480a00713221b5149dd411a7d0f329a4b28f71846f9: Status 404 returned error can't find the container with id f13eb7e1f0250646b37fb480a00713221b5149dd411a7d0f329a4b28f71846f9
Sep 17 15:47:25.098074 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.097969    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" event={"ID":"fdc4b642-4115-4c21-85fb-2967bebdbd59","Type":"ContainerStarted","Data":"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"}
Sep 17 15:47:25.098074 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.098013    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" event={"ID":"fdc4b642-4115-4c21-85fb-2967bebdbd59","Type":"ContainerStarted","Data":"f13eb7e1f0250646b37fb480a00713221b5149dd411a7d0f329a4b28f71846f9"}
Sep 17 15:47:25.099412 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.099392    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"
Sep 17 15:47:25.099412 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.099400    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r" event={"ID":"fd8f42d2-37cf-4680-8e35-79136a4b3be1","Type":"ContainerDied","Data":"7762d274d07df9e61878ea01279cc6d8288d00d9cf4e1215f94885ab616c6a03"}
Sep 17 15:47:25.099552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.099437    2564 scope.go:117] "RemoveContainer" containerID="a1359d991094a1a6918ebbcfe3002686313efde1080c359ed09fec2bf0bf5b4e"
Sep 17 15:47:25.137386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.137330    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" podStartSLOduration=1.137313507 podStartE2EDuration="1.137313507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:25.136006703 +0000 UTC m=+1037.103184522" watchObservedRunningTime="2026-09-17 15:47:25.137313507 +0000 UTC m=+1037.104491322"
Sep 17 15:47:25.163577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.163549    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:25.175085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:25.175049    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-5b08fa/explicit-model-5b08fa-kserve-7cb5995469-l8s2r"]
Sep 17 15:47:26.527168 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:26.527137    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd8f42d2-37cf-4680-8e35-79136a4b3be1" path="/var/lib/kubelet/pods/fd8f42d2-37cf-4680-8e35-79136a4b3be1/volumes"
Sep 17 15:47:29.742000 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:29.741965    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:47:29.742370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:29.742162    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy" containerID="cri-o://7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec" gracePeriod=30
Sep 17 15:47:32.613396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:32.613358    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:47:34.559565 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:34.559532    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:34.560066 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:34.559582    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:34.572196 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:34.572166    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:35.015651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.015603    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:47:35.154172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154093    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154131    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154158    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154295    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154329    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154423 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154378    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154452    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154485    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vdrk\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154514    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data\") pod \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\" (UID: \"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb\") "
Sep 17 15:47:35.154595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154534    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.154837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154682    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.154837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154764    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.154988 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154965    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.155056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.154991    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.155056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.155011    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.155056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.155026    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.155056 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.155025    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:47:35.156628 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.156567    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:35.156753 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.156606    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:47:35.156753 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.156637    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk" (OuterVolumeSpecName: "kube-api-access-8vdrk") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "kube-api-access-8vdrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:47:35.156869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.156761    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" (UID: "8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:47:35.163113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.163027    2564 generic.go:356] "Generic (PLEG): container finished" podID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerID="7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec" exitCode=0
Sep 17 15:47:35.163113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.163078    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"
Sep 17 15:47:35.163113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.163098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerDied","Data":"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"}
Sep 17 15:47:35.163269 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.163179    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n" event={"ID":"8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb","Type":"ContainerDied","Data":"b10955d274561facfc82d42ad85c798ef5a7b2d426d50c5f819dc7764565544b"}
Sep 17 15:47:35.163269 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.163209    2564 scope.go:117] "RemoveContainer" containerID="7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"
Sep 17 15:47:35.172651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.172633    2564 scope.go:117] "RemoveContainer" containerID="df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"
Sep 17 15:47:35.178077 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.178055    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:47:35.181420 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.181402    2564 scope.go:117] "RemoveContainer" containerID="7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"
Sep 17 15:47:35.181692 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:47:35.181673    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec\": container with ID starting with 7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec not found: ID does not exist" containerID="7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"
Sep 17 15:47:35.181750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.181699    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec"} err="failed to get container status \"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec\": rpc error: code = NotFound desc = could not find container \"7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec\": container with ID starting with 7aeb20841bac1e1046ed52a2cd6f56e4bf5d1b861d9912d3644d8fb43eeab3ec not found: ID does not exist"
Sep 17 15:47:35.181750 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.181715    2564 scope.go:117] "RemoveContainer" containerID="df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"
Sep 17 15:47:35.181964 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:47:35.181948    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f\": container with ID starting with df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f not found: ID does not exist" containerID="df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"
Sep 17 15:47:35.182014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.181967    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f"} err="failed to get container status \"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f\": rpc error: code = NotFound desc = could not find container \"df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f\": container with ID starting with df76ef90450622b507e0acbaaed7b64472f0ec52599736129d68878bb19d8b5f not found: ID does not exist"
Sep 17 15:47:35.190337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.190316    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:47:35.194033 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.194014    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-5b08fa-openshift-default-7bffc6fc6-4z47n"]
Sep 17 15:47:35.256400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256356    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.256400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256384    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.256400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256397    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.256400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256409    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.256400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256422    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vdrk\" (UniqueName: \"kubernetes.io/projected/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-kube-api-access-8vdrk\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:35.256746 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:35.256434    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:47:36.525970 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:36.525929    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" path="/var/lib/kubelet/pods/8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb/volumes"
Sep 17 15:47:43.579768 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.579736    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:47:43.580188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.580141    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.580188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.580153    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.580262 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.580235    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.584551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.584527    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:47:43.584695 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.584678    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.585138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.585116    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.585138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.585138    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.585275 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.585205    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e04f0b2-23b9-4c5e-9d1a-332e9d6f83bb" containerName="istio-proxy"
Sep 17 15:47:43.587162 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.587143    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-70207142\""
Sep 17 15:47:43.588103 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.588086    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.612661 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.610020    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:47:43.613922 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.613897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:47:43.742445 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.742391    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.742657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.742579    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hph\" (UniqueName: \"kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.742738 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.742714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.742862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.742834    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggls5\" (UniqueName: \"kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.844055 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.843967    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hph\" (UniqueName: \"kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.844055 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.844019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.844279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.844070    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggls5\" (UniqueName: \"kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.844279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.844107    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.844798 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.844779    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.844844 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.844799    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.854583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.854559    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hph\" (UniqueName: \"kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph\") pod \"payload-processing-e2e-mt-70207142-6655594565-tgrt9\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") " pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.855233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.855215    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggls5\" (UniqueName: \"kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5\") pod \"payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:43.898088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.898055    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:43.914396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:43.914366    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:44.185425 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:44.185396    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcf405f_5c28_4438_b965_7a8dbfc1d09d.slice/crio-6cce07e1288a994676b29ada6a21582b4f757c782292f7a3d433bff5f08050b5 WatchSource:0}: Error finding container 6cce07e1288a994676b29ada6a21582b4f757c782292f7a3d433bff5f08050b5: Status 404 returned error can't find the container with id 6cce07e1288a994676b29ada6a21582b4f757c782292f7a3d433bff5f08050b5
Sep 17 15:47:44.185531 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:44.185507    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:47:44.199810 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:44.199784    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:47:44.201927 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:47:44.201901    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319ba7a6_f4a6_4a48_bce4_b4d64179da7f.slice/crio-e6ee5216c425d8eca3f8f559971b1a00e224ff44c34e8606481457139e2b344a WatchSource:0}: Error finding container e6ee5216c425d8eca3f8f559971b1a00e224ff44c34e8606481457139e2b344a: Status 404 returned error can't find the container with id e6ee5216c425d8eca3f8f559971b1a00e224ff44c34e8606481457139e2b344a
Sep 17 15:47:44.219564 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:44.219526    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" event={"ID":"319ba7a6-f4a6-4a48-bce4-b4d64179da7f","Type":"ContainerStarted","Data":"e6ee5216c425d8eca3f8f559971b1a00e224ff44c34e8606481457139e2b344a"}
Sep 17 15:47:44.221283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:44.221258    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" event={"ID":"1fcf405f-5c28-4438-b965-7a8dbfc1d09d","Type":"ContainerStarted","Data":"6cce07e1288a994676b29ada6a21582b4f757c782292f7a3d433bff5f08050b5"}
Sep 17 15:47:45.229633 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.229570    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" event={"ID":"319ba7a6-f4a6-4a48-bce4-b4d64179da7f","Type":"ContainerStarted","Data":"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"}
Sep 17 15:47:45.230097 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.229667    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:47:45.231352 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.231328    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" event={"ID":"1fcf405f-5c28-4438-b965-7a8dbfc1d09d","Type":"ContainerStarted","Data":"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"}
Sep 17 15:47:45.231467 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.231382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:45.251313 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.251262    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" podStartSLOduration=2.2512474510000002 podStartE2EDuration="2.251247451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:45.249153909 +0000 UTC m=+1057.216331727" watchObservedRunningTime="2026-09-17 15:47:45.251247451 +0000 UTC m=+1057.218425269"
Sep 17 15:47:45.310947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:45.310887    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" podStartSLOduration=2.310868914 podStartE2EDuration="2.310868914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:47:45.306283702 +0000 UTC m=+1057.273461523" watchObservedRunningTime="2026-09-17 15:47:45.310868914 +0000 UTC m=+1057.278046731"
Sep 17 15:47:56.239099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:56.239062    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:47:56.239583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:47:56.239146    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:48:07.093305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:07.093232    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:48:16.861931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.861898    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"]
Sep 17 15:48:16.867138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.867117    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:16.870388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.870364    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:48:16.870495 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.870393    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:48:16.877628 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.877592    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"]
Sep 17 15:48:16.993080 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:16.993040    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-70207142-skhjt\" (UID: \"33fdfaa4-61a9-44f5-8bbc-de7d316e230b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:17.094546 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:17.094503    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-70207142-skhjt\" (UID: \"33fdfaa4-61a9-44f5-8bbc-de7d316e230b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:17.095209 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:17.095180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-70207142-skhjt\" (UID: \"33fdfaa4-61a9-44f5-8bbc-de7d316e230b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:17.179092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:17.179014    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:17.469626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:17.469584    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"]
Sep 17 15:48:17.471684 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:48:17.471653    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33fdfaa4_61a9_44f5_8bbc_de7d316e230b.slice/crio-34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60 WatchSource:0}: Error finding container 34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60: Status 404 returned error can't find the container with id 34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60
Sep 17 15:48:18.435320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:18.435283    2564 generic.go:356] "Generic (PLEG): container finished" podID="33fdfaa4-61a9-44f5-8bbc-de7d316e230b" containerID="e4ae531c92bf07b133b536e40bb70799893de48c4f08f2aad8029135ce83a4f9" exitCode=0
Sep 17 15:48:18.435745 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:18.435359    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt" event={"ID":"33fdfaa4-61a9-44f5-8bbc-de7d316e230b","Type":"ContainerDied","Data":"e4ae531c92bf07b133b536e40bb70799893de48c4f08f2aad8029135ce83a4f9"}
Sep 17 15:48:18.435745 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:18.435458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt" event={"ID":"33fdfaa4-61a9-44f5-8bbc-de7d316e230b","Type":"ContainerStarted","Data":"34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60"}
Sep 17 15:48:19.575372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:19.575348    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:19.727176 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:19.727089    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca\") pod \"33fdfaa4-61a9-44f5-8bbc-de7d316e230b\" (UID: \"33fdfaa4-61a9-44f5-8bbc-de7d316e230b\") "
Sep 17 15:48:19.727479 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:19.727456    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33fdfaa4-61a9-44f5-8bbc-de7d316e230b" (UID: "33fdfaa4-61a9-44f5-8bbc-de7d316e230b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:19.828801 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:19.828767    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33fdfaa4-61a9-44f5-8bbc-de7d316e230b-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:20.448758 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:20.448728    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"
Sep 17 15:48:20.448930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:20.448772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt" event={"ID":"33fdfaa4-61a9-44f5-8bbc-de7d316e230b","Type":"ContainerDied","Data":"34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60"}
Sep 17 15:48:20.448930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:20.448820    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34223a35b9870b22ba1fe791bff1e32d6833d869a2ec1afbd97c01e34784ed60"
Sep 17 15:48:25.888400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:25.888364    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:48:27.763355 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:27.763309    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:48:27.763979 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:27.763657    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" podUID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" containerName="payload-processing" containerID="cri-o://8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86" gracePeriod=30
Sep 17 15:48:27.779235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:27.779204    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:48:27.779436 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:27.779400    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" podUID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" containerName="payload-pre-processing" containerID="cri-o://f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5" gracePeriod=30
Sep 17 15:48:28.028327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.028244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"]
Sep 17 15:48:28.032706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.032679    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-70207142-skhjt"]
Sep 17 15:48:28.073945 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.073922    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:48:28.081657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.081630    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:48:28.108072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108033    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume\") pod \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") "
Sep 17 15:48:28.108072 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108085    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5hph\" (UniqueName: \"kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph\") pod \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") "
Sep 17 15:48:28.108350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108134    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggls5\" (UniqueName: \"kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5\") pod \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\" (UID: \"319ba7a6-f4a6-4a48-bce4-b4d64179da7f\") "
Sep 17 15:48:28.108350 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108177    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume\") pod \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\" (UID: \"1fcf405f-5c28-4438-b965-7a8dbfc1d09d\") "
Sep 17 15:48:28.108470 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108433    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "319ba7a6-f4a6-4a48-bce4-b4d64179da7f" (UID: "319ba7a6-f4a6-4a48-bce4-b4d64179da7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:28.108530 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108515    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.108761 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.108732    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1fcf405f-5c28-4438-b965-7a8dbfc1d09d" (UID: "1fcf405f-5c28-4438-b965-7a8dbfc1d09d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:28.110565 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.110533    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5" (OuterVolumeSpecName: "kube-api-access-ggls5") pod "319ba7a6-f4a6-4a48-bce4-b4d64179da7f" (UID: "319ba7a6-f4a6-4a48-bce4-b4d64179da7f"). InnerVolumeSpecName "kube-api-access-ggls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:28.111311 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.111281    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph" (OuterVolumeSpecName: "kube-api-access-r5hph") pod "1fcf405f-5c28-4438-b965-7a8dbfc1d09d" (UID: "1fcf405f-5c28-4438-b965-7a8dbfc1d09d"). InnerVolumeSpecName "kube-api-access-r5hph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:48:28.209957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.209925    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hph\" (UniqueName: \"kubernetes.io/projected/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-kube-api-access-r5hph\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.209957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.209957    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggls5\" (UniqueName: \"kubernetes.io/projected/319ba7a6-f4a6-4a48-bce4-b4d64179da7f-kube-api-access-ggls5\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.209957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.209971    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fcf405f-5c28-4438-b965-7a8dbfc1d09d-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:28.495305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.495204    2564 generic.go:356] "Generic (PLEG): container finished" podID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" containerID="f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5" exitCode=0
Sep 17 15:48:28.495305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.495252    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"
Sep 17 15:48:28.495305 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.495285    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" event={"ID":"319ba7a6-f4a6-4a48-bce4-b4d64179da7f","Type":"ContainerDied","Data":"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"}
Sep 17 15:48:28.495593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.495334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf" event={"ID":"319ba7a6-f4a6-4a48-bce4-b4d64179da7f","Type":"ContainerDied","Data":"e6ee5216c425d8eca3f8f559971b1a00e224ff44c34e8606481457139e2b344a"}
Sep 17 15:48:28.495593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.495358    2564 scope.go:117] "RemoveContainer" containerID="f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"
Sep 17 15:48:28.496795 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.496769    2564 generic.go:356] "Generic (PLEG): container finished" podID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" containerID="8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86" exitCode=0
Sep 17 15:48:28.496907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.496807    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" event={"ID":"1fcf405f-5c28-4438-b965-7a8dbfc1d09d","Type":"ContainerDied","Data":"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"}
Sep 17 15:48:28.496907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.496863    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9" event={"ID":"1fcf405f-5c28-4438-b965-7a8dbfc1d09d","Type":"ContainerDied","Data":"6cce07e1288a994676b29ada6a21582b4f757c782292f7a3d433bff5f08050b5"}
Sep 17 15:48:28.496907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.496816    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"
Sep 17 15:48:28.507546 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.507503    2564 scope.go:117] "RemoveContainer" containerID="f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"
Sep 17 15:48:28.507935 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:48:28.507902    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5\": container with ID starting with f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5 not found: ID does not exist" containerID="f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"
Sep 17 15:48:28.508086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.507937    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5"} err="failed to get container status \"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5\": rpc error: code = NotFound desc = could not find container \"f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5\": container with ID starting with f693dac7a7aeac03731462b54211e960aa256134c320631e7d615c4eaada04a5 not found: ID does not exist"
Sep 17 15:48:28.508086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.507962    2564 scope.go:117] "RemoveContainer" containerID="8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"
Sep 17 15:48:28.528406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.528335    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33fdfaa4-61a9-44f5-8bbc-de7d316e230b" path="/var/lib/kubelet/pods/33fdfaa4-61a9-44f5-8bbc-de7d316e230b/volumes"
Sep 17 15:48:28.528406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.528358    2564 scope.go:117] "RemoveContainer" containerID="8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"
Sep 17 15:48:28.528737 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:48:28.528719    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86\": container with ID starting with 8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86 not found: ID does not exist" containerID="8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"
Sep 17 15:48:28.528784 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.528744    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86"} err="failed to get container status \"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86\": rpc error: code = NotFound desc = could not find container \"8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86\": container with ID starting with 8567f8f15350e39fe7e59f0856656b135e515bf6e553086d3bd8668a2708da86 not found: ID does not exist"
Sep 17 15:48:28.542375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.542342    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:48:28.555426 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.555394    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-70207142-d64f9fdbb-lhrlf"]
Sep 17 15:48:28.575907 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.575874    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:48:28.578003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:28.577978    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-70207142-6655594565-tgrt9"]
Sep 17 15:48:30.525492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:30.525455    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" path="/var/lib/kubelet/pods/1fcf405f-5c28-4438-b965-7a8dbfc1d09d/volumes"
Sep 17 15:48:30.525915 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:30.525855    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" path="/var/lib/kubelet/pods/319ba7a6-f4a6-4a48-bce4-b4d64179da7f/volumes"
Sep 17 15:48:40.864540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864501    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864949    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" containerName="payload-pre-processing"
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864963    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" containerName="payload-pre-processing"
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864975    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" containerName="payload-processing"
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864981    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" containerName="payload-processing"
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.864996    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33fdfaa4-61a9-44f5-8bbc-de7d316e230b" containerName="revoke-keys"
Sep 17 15:48:40.865048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.865003    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="33fdfaa4-61a9-44f5-8bbc-de7d316e230b" containerName="revoke-keys"
Sep 17 15:48:40.865364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.865072    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="319ba7a6-f4a6-4a48-bce4-b4d64179da7f" containerName="payload-pre-processing"
Sep 17 15:48:40.865364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.865084    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcf405f-5c28-4438-b965-7a8dbfc1d09d" containerName="payload-processing"
Sep 17 15:48:40.865364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.865095    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="33fdfaa4-61a9-44f5-8bbc-de7d316e230b" containerName="revoke-keys"
Sep 17 15:48:40.868428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.868407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.871164 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.871146    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d9afa3\"/\"kube-root-ca.crt\""
Sep 17 15:48:40.871349 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.871330    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-d9afa3\"/\"openshift-service-ca.crt\""
Sep 17 15:48:40.871434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.871365    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d9afa3\"/\"default-dockercfg-m7pgm\""
Sep 17 15:48:40.871822 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.871809    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-d9afa3\"/\"noten-model-d9afa3-kserve-self-signed-certs\""
Sep 17 15:48:40.886886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.886855    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:48:40.940121 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940090    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.940266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940135    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.940266 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940197    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.940345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940269    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.940345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940303    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm6x8\" (UniqueName: \"kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:40.940345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:40.940330    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042000 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.041917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042011    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042051    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042104    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm6x8\" (UniqueName: \"kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042146    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.042626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.042378    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.043127 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.043096    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.043247 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.043209    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.045311 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.045263    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.046032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.046011    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.056551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.056483    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm6x8\" (UniqueName: \"kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8\") pod \"noten-model-d9afa3-kserve-8fc4647cf-vtjdv\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") " pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.187404 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.187324    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:41.470776 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.470747    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:48:41.472954 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:48:41.472929    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c68d398_4223_4873_9532_7034966e54bc.slice/crio-1d3d26ed84fdc9cf123a99c4a9852092db056a4c27c626662744f1eef5ec49cd WatchSource:0}: Error finding container 1d3d26ed84fdc9cf123a99c4a9852092db056a4c27c626662744f1eef5ec49cd: Status 404 returned error can't find the container with id 1d3d26ed84fdc9cf123a99c4a9852092db056a4c27c626662744f1eef5ec49cd
Sep 17 15:48:41.586999 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.586961    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" event={"ID":"2c68d398-4223-4873-9532-7034966e54bc","Type":"ContainerStarted","Data":"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"}
Sep 17 15:48:41.587149 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.587014    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" event={"ID":"2c68d398-4223-4873-9532-7034966e54bc","Type":"ContainerStarted","Data":"1d3d26ed84fdc9cf123a99c4a9852092db056a4c27c626662744f1eef5ec49cd"}
Sep 17 15:48:41.610963 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:41.610914    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" podStartSLOduration=1.6108979209999998 podStartE2EDuration="1.610897921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:41.608954228 +0000 UTC m=+1113.576132049" watchObservedRunningTime="2026-09-17 15:48:41.610897921 +0000 UTC m=+1113.578075739"
Sep 17 15:48:42.985923 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:42.985892    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"]
Sep 17 15:48:42.989406 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:42.989386    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:42.993450 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:42.993427    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:48:42.993597 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:42.993580    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:48:43.006249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.006226    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"]
Sep 17 15:48:43.063926 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.063892    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr\" (UID: \"99db6872-12b3-45ff-bd77-70cecdeba62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:43.165285 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.165254    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr\" (UID: \"99db6872-12b3-45ff-bd77-70cecdeba62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:43.165910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.165893    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr\" (UID: \"99db6872-12b3-45ff-bd77-70cecdeba62e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:43.300675 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.300639    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:43.594128 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.594106    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"]
Sep 17 15:48:43.595838 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:48:43.595809    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99db6872_12b3_45ff_bd77_70cecdeba62e.slice/crio-963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204 WatchSource:0}: Error finding container 963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204: Status 404 returned error can't find the container with id 963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204
Sep 17 15:48:43.603997 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:43.603972    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr" event={"ID":"99db6872-12b3-45ff-bd77-70cecdeba62e","Type":"ContainerStarted","Data":"963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204"}
Sep 17 15:48:44.611639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:44.611532    2564 generic.go:356] "Generic (PLEG): container finished" podID="99db6872-12b3-45ff-bd77-70cecdeba62e" containerID="7497f630a349e4ada719245abda0089fe9b15f185a37cc873f6e0a338baa980b" exitCode=0
Sep 17 15:48:44.611986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:44.611641    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr" event={"ID":"99db6872-12b3-45ff-bd77-70cecdeba62e","Type":"ContainerDied","Data":"7497f630a349e4ada719245abda0089fe9b15f185a37cc873f6e0a338baa980b"}
Sep 17 15:48:45.768331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:45.768306    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:45.896502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:45.896428    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca\") pod \"99db6872-12b3-45ff-bd77-70cecdeba62e\" (UID: \"99db6872-12b3-45ff-bd77-70cecdeba62e\") "
Sep 17 15:48:45.896792 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:45.896769    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99db6872-12b3-45ff-bd77-70cecdeba62e" (UID: "99db6872-12b3-45ff-bd77-70cecdeba62e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:48:45.997948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:45.997918    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99db6872-12b3-45ff-bd77-70cecdeba62e-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:48:46.629740 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:46.629709    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"
Sep 17 15:48:46.629886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:46.629742    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr" event={"ID":"99db6872-12b3-45ff-bd77-70cecdeba62e","Type":"ContainerDied","Data":"963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204"}
Sep 17 15:48:46.629886 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:46.629793    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="963b28bafb2244ae4e198d23b347d8370b63c6a11c818f931e5b474a54588204"
Sep 17 15:48:51.188529 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:51.188488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:51.189140 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:51.188551    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:51.201441 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:51.201411    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:51.487251 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:51.487163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:48:51.673941 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:51.673907    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:48:52.695490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.695456    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:48:52.696100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.696082    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99db6872-12b3-45ff-bd77-70cecdeba62e" containerName="revoke-keys"
Sep 17 15:48:52.696161 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.696103    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="99db6872-12b3-45ff-bd77-70cecdeba62e" containerName="revoke-keys"
Sep 17 15:48:52.696206 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.696188    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="99db6872-12b3-45ff-bd77-70cecdeba62e" containerName="revoke-keys"
Sep 17 15:48:52.699386 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.699364    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.703740 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.703714    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-2b23c7-openshift-default-dockercfg-k4qks\""
Sep 17 15:48:52.724650 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.724596    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:48:52.770478 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770448    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770490    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dsm\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770578    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770644    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770818 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770748    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770814    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770841    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770910    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.770948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.770940    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.871702 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871670    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.871702 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871706    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.871912 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871837    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.871912 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871871    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872006 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871917    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872006 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.871968    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dsm\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872059    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872113 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872090    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872116    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872133    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872159    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872255    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872482 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872455    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.872905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.872883    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.874242 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.874214    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.874421 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.874403    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.882514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.882494    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:52.892046 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:52.892014    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dsm\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm\") pod \"e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") " pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:53.009910 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.009877    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:53.298961 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:48:53.298924    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9243c8a3_eb3c_49c4_bdc6_906984e7af60.slice/crio-9a422c0be0c39fa3b713279f6107907374a9a1ab54a70e0d764fd67520a505f3 WatchSource:0}: Error finding container 9a422c0be0c39fa3b713279f6107907374a9a1ab54a70e0d764fd67520a505f3: Status 404 returned error can't find the container with id 9a422c0be0c39fa3b713279f6107907374a9a1ab54a70e0d764fd67520a505f3
Sep 17 15:48:53.299179 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.299159    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:48:53.301207 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.301180    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:48:53.301280 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.301239    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:48:53.301280 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.301266    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:48:53.545876 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.545798    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:48:53.683099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.683054    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerStarted","Data":"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"}
Sep 17 15:48:53.683270 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.683184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerStarted","Data":"9a422c0be0c39fa3b713279f6107907374a9a1ab54a70e0d764fd67520a505f3"}
Sep 17 15:48:53.684567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.684543    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"]
Sep 17 15:48:53.686870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.686846    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-449c47-29827665-96lgf"]
Sep 17 15:48:53.712195 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.712145    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podStartSLOduration=1.712124966 podStartE2EDuration="1.712124966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:53.708578444 +0000 UTC m=+1125.675756290" watchObservedRunningTime="2026-09-17 15:48:53.712124966 +0000 UTC m=+1125.679302785"
Sep 17 15:48:53.989155 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.989070    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"]
Sep 17 15:48:53.995385 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:53.995348    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-449c47-6zmtr"]
Sep 17 15:48:54.010312 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.010288    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:54.015449 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.015418    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:54.270007 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.269980    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:48:54.525107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.525034    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f887bf0-c4df-4579-9126-7b87808584fe" path="/var/lib/kubelet/pods/1f887bf0-c4df-4579-9126-7b87808584fe/volumes"
Sep 17 15:48:54.525351 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.525337    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99db6872-12b3-45ff-bd77-70cecdeba62e" path="/var/lib/kubelet/pods/99db6872-12b3-45ff-bd77-70cecdeba62e/volumes"
Sep 17 15:48:54.689415 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.689380    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:54.690401 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:54.690382    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:48:56.670150 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:56.670120    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:48:59.032129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.032093    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:48:59.036293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.036264    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.039470 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.039449    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-2b23c7-dockercfg-q6nmr\""
Sep 17 15:48:59.039947 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.039931    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-2b23c7\""
Sep 17 15:48:59.047762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.047741    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:48:59.137985 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.137941    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk6dq\" (UniqueName: \"kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.138341 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.138319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.241734 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.241699    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk6dq\" (UniqueName: \"kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.241734 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.241739    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.242418 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.242399    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.250426 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.250404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk6dq\" (UniqueName: \"kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq\") pod \"payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.348086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.348016    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.640390 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.640269    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:48:59.648901 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:48:59.648866    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda726c89e_61ff_4e83_8ecc_6fb8b13bdfba.slice/crio-9b6574cad94ba0d4ea484e384aa23127b01099be45ce71ea89ac1117ccb0c55c WatchSource:0}: Error finding container 9b6574cad94ba0d4ea484e384aa23127b01099be45ce71ea89ac1117ccb0c55c: Status 404 returned error can't find the container with id 9b6574cad94ba0d4ea484e384aa23127b01099be45ce71ea89ac1117ccb0c55c
Sep 17 15:48:59.722057 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.722023    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" event={"ID":"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba","Type":"ContainerStarted","Data":"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"}
Sep 17 15:48:59.722185 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.722084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" event={"ID":"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba","Type":"ContainerStarted","Data":"9b6574cad94ba0d4ea484e384aa23127b01099be45ce71ea89ac1117ccb0c55c"}
Sep 17 15:48:59.722257 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.722209    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:48:59.746641 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:48:59.744210    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" podStartSLOduration=0.744190484 podStartE2EDuration="744.190484ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:59.741349771 +0000 UTC m=+1131.708527588" watchObservedRunningTime="2026-09-17 15:48:59.744190484 +0000 UTC m=+1131.711368303"
Sep 17 15:49:08.565562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.565509    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" podUID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" containerName="main" containerID="cri-o://5f4b47fb209981f24db9fd897c121b8a727648bc0ae562a7d043585c3913330f" gracePeriod=45
Sep 17 15:49:08.586980 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.586948    2564 scope.go:117] "RemoveContainer" containerID="894798d62a5b3eb8e894ffac620a5b371d9dc1a80dc7bb2bc8d39ba41d52e469"
Sep 17 15:49:08.677235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.677215    2564 scope.go:117] "RemoveContainer" containerID="e581a705a37841f351e606e43ef5ff1ab7ffee1b699ca5bf71019f9db21f1535"
Sep 17 15:49:08.777721 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.777695    2564 generic.go:356] "Generic (PLEG): container finished" podID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" containerID="5f4b47fb209981f24db9fd897c121b8a727648bc0ae562a7d043585c3913330f" exitCode=0
Sep 17 15:49:08.777911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.777772    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" event={"ID":"ab21152d-21df-4d71-bd8b-0da3bc0bc354","Type":"ContainerDied","Data":"5f4b47fb209981f24db9fd897c121b8a727648bc0ae562a7d043585c3913330f"}
Sep 17 15:49:08.800955 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.800925    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:49:08.831966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.831907    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:49:08.951053 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951009    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.951255 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951098    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.951255 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951141    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.951255 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951222    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsnrh\" (UniqueName: \"kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.951255 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951246    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.951478 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951317    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir\") pod \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\" (UID: \"ab21152d-21df-4d71-bd8b-0da3bc0bc354\") "
Sep 17 15:49:08.952670 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.951885    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:08.952670 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.952044    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:08.953075 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.953042    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home" (OuterVolumeSpecName: "home") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:08.956480 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.954720    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm" (OuterVolumeSpecName: "dshm") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:08.957003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.956741    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:08.957003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:08.956817    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh" (OuterVolumeSpecName: "kube-api-access-tsnrh") pod "ab21152d-21df-4d71-bd8b-0da3bc0bc354" (UID: "ab21152d-21df-4d71-bd8b-0da3bc0bc354"). InnerVolumeSpecName "kube-api-access-tsnrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:09.052920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052888    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.052920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052918    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.053083 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052929    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.053083 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052938    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnrh\" (UniqueName: \"kubernetes.io/projected/ab21152d-21df-4d71-bd8b-0da3bc0bc354-kube-api-access-tsnrh\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.053083 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052946    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.053083 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.052955    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab21152d-21df-4d71-bd8b-0da3bc0bc354-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.288967 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.288930    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" podUID="fdc4b642-4115-4c21-85fb-2967bebdbd59" containerName="main" containerID="cri-o://838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955" gracePeriod=45
Sep 17 15:49:09.537639 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.537592    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:49:09.658768 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.658742    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.658803    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.658907    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zrtz\" (UniqueName: \"kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.658946    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.658970    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.659034    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home" (OuterVolumeSpecName: "home") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:09.659107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.659080    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache\") pod \"fdc4b642-4115-4c21-85fb-2967bebdbd59\" (UID: \"fdc4b642-4115-4c21-85fb-2967bebdbd59\") "
Sep 17 15:49:09.659434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.659180    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:09.659434 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.659345    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache" (OuterVolumeSpecName: "model-cache") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:09.660047 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.659981    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.660047 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.660007    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.660047 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.660023    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.661340 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.661315    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm" (OuterVolumeSpecName: "dshm") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:09.661433 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.661395    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz" (OuterVolumeSpecName: "kube-api-access-8zrtz") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "kube-api-access-8zrtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:09.661476 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.661446    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fdc4b642-4115-4c21-85fb-2967bebdbd59" (UID: "fdc4b642-4115-4c21-85fb-2967bebdbd59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:09.761224 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.761202    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fdc4b642-4115-4c21-85fb-2967bebdbd59-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.761327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.761226    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zrtz\" (UniqueName: \"kubernetes.io/projected/fdc4b642-4115-4c21-85fb-2967bebdbd59-kube-api-access-8zrtz\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.761327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.761237    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fdc4b642-4115-4c21-85fb-2967bebdbd59-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:09.785307 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.785287    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"
Sep 17 15:49:09.785419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.785334    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p" event={"ID":"ab21152d-21df-4d71-bd8b-0da3bc0bc354","Type":"ContainerDied","Data":"d3d576f3afa5b9faa98189ec13e1861ca548dbbe5dc65e2025b5fea177067772"}
Sep 17 15:49:09.785419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.785389    2564 scope.go:117] "RemoveContainer" containerID="5f4b47fb209981f24db9fd897c121b8a727648bc0ae562a7d043585c3913330f"
Sep 17 15:49:09.786963 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.786899    2564 generic.go:356] "Generic (PLEG): container finished" podID="fdc4b642-4115-4c21-85fb-2967bebdbd59" containerID="838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955" exitCode=0
Sep 17 15:49:09.786963 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.786924    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" event={"ID":"fdc4b642-4115-4c21-85fb-2967bebdbd59","Type":"ContainerDied","Data":"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"}
Sep 17 15:49:09.786963 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.786939    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"
Sep 17 15:49:09.787145 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.786975    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww" event={"ID":"fdc4b642-4115-4c21-85fb-2967bebdbd59","Type":"ContainerDied","Data":"f13eb7e1f0250646b37fb480a00713221b5149dd411a7d0f329a4b28f71846f9"}
Sep 17 15:49:09.796371 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.796353    2564 scope.go:117] "RemoveContainer" containerID="838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"
Sep 17 15:49:09.804257 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.804243    2564 scope.go:117] "RemoveContainer" containerID="838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"
Sep 17 15:49:09.804506 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:49:09.804489    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955\": container with ID starting with 838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955 not found: ID does not exist" containerID="838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"
Sep 17 15:49:09.804550 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.804515    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955"} err="failed to get container status \"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955\": rpc error: code = NotFound desc = could not find container \"838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955\": container with ID starting with 838f7ea7320ce4b2fd9de0dab39932900dc1ff254e6b98b9c105bd0a2e1f3955 not found: ID does not exist"
Sep 17 15:49:09.814300 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.814278    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:49:09.822259 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.822240    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a2dc23/test-model-a2dc23-kserve-75b5b8bf55-44m2p"]
Sep 17 15:49:09.834446 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.834395    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:49:09.842539 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:09.842513    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-b-bd4c6f71-kserve-5d54ddcd6b-rgnww"]
Sep 17 15:49:10.528153 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:10.528119    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" path="/var/lib/kubelet/pods/ab21152d-21df-4d71-bd8b-0da3bc0bc354/volumes"
Sep 17 15:49:10.528719 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:10.528689    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdc4b642-4115-4c21-85fb-2967bebdbd59" path="/var/lib/kubelet/pods/fdc4b642-4115-4c21-85fb-2967bebdbd59/volumes"
Sep 17 15:49:10.660970 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:10.659263    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:49:10.729768 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:10.729742    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:49:11.688560 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:11.688512    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" podUID="2c68d398-4223-4873-9532-7034966e54bc" containerName="main" containerID="cri-o://d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b" gracePeriod=45
Sep 17 15:49:11.980363 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:11.980338    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:49:12.081837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.081809    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.081856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.081919    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.081940    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.081965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm6x8\" (UniqueName: \"kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082231 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082019    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs\") pod \"2c68d398-4223-4873-9532-7034966e54bc\" (UID: \"2c68d398-4223-4873-9532-7034966e54bc\") "
Sep 17 15:49:12.082231 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082120    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:12.082231 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082176    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home" (OuterVolumeSpecName: "home") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:12.082392 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082316    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:12.082506 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082487    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.082584 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082512    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.082584 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.082523    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.084309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.084282    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm" (OuterVolumeSpecName: "dshm") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:12.084433 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.084378    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:12.084433 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.084410    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8" (OuterVolumeSpecName: "kube-api-access-xm6x8") pod "2c68d398-4223-4873-9532-7034966e54bc" (UID: "2c68d398-4223-4873-9532-7034966e54bc"). InnerVolumeSpecName "kube-api-access-xm6x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:12.184146 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.184115    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2c68d398-4223-4873-9532-7034966e54bc-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.184146 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.184143    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm6x8\" (UniqueName: \"kubernetes.io/projected/2c68d398-4223-4873-9532-7034966e54bc-kube-api-access-xm6x8\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.184339 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.184156    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c68d398-4223-4873-9532-7034966e54bc-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:12.809389 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.809298    2564 generic.go:356] "Generic (PLEG): container finished" podID="2c68d398-4223-4873-9532-7034966e54bc" containerID="d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b" exitCode=0
Sep 17 15:49:12.810066 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.810039    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" event={"ID":"2c68d398-4223-4873-9532-7034966e54bc","Type":"ContainerDied","Data":"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"}
Sep 17 15:49:12.810217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.810201    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv" event={"ID":"2c68d398-4223-4873-9532-7034966e54bc","Type":"ContainerDied","Data":"1d3d26ed84fdc9cf123a99c4a9852092db056a4c27c626662744f1eef5ec49cd"}
Sep 17 15:49:12.810330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.810317    2564 scope.go:117] "RemoveContainer" containerID="d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"
Sep 17 15:49:12.810571 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.810554    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"
Sep 17 15:49:12.824429 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.824399    2564 scope.go:117] "RemoveContainer" containerID="d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"
Sep 17 15:49:12.824945 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:49:12.824918    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b\": container with ID starting with d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b not found: ID does not exist" containerID="d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"
Sep 17 15:49:12.825093 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.825071    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b"} err="failed to get container status \"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b\": rpc error: code = NotFound desc = could not find container \"d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b\": container with ID starting with d10ec8449b158cb22870ada9154944f0c1c7c3053393a8e140c0573beded986b not found: ID does not exist"
Sep 17 15:49:12.837142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.837118    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:49:12.839168 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:12.839148    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-d9afa3/noten-model-d9afa3-kserve-8fc4647cf-vtjdv"]
Sep 17 15:49:14.530486 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:14.530448    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c68d398-4223-4873-9532-7034966e54bc" path="/var/lib/kubelet/pods/2c68d398-4223-4873-9532-7034966e54bc/volumes"
Sep 17 15:49:20.873458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.873426    2564 generic.go:356] "Generic (PLEG): container finished" podID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerID="c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17" exitCode=0
Sep 17 15:49:20.873913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.873497    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerDied","Data":"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"}
Sep 17 15:49:20.873972 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.873955    2564 scope.go:117] "RemoveContainer" containerID="c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"
Sep 17 15:49:20.875148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.875118    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:49:20.875249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.875204    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:49:20.875314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:20.875256    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:49:21.881471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:21.881433    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerStarted","Data":"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"}
Sep 17 15:49:22.010286 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:22.010252    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:49:22.011541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:22.011504    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:22.011744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:22.011579    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:23.010782 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:23.010740    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:23.011258 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:23.010803    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:24.010608 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:24.010557    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:24.011021 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:24.010974    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:24.689665 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:24.689606    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:49:25.011302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:25.011266    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:25.011904 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:25.011869    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:25.682575 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:25.682530    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" podUID="496cd31b-be97-4723-a260-886153fc3359" containerName="main" containerID="cri-o://60fc4042f3ba0a71913dfce23850010edec356954ee41cc44c62f52f7e8be135" gracePeriod=45
Sep 17 15:49:25.939706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:25.939604    2564 generic.go:356] "Generic (PLEG): container finished" podID="496cd31b-be97-4723-a260-886153fc3359" containerID="60fc4042f3ba0a71913dfce23850010edec356954ee41cc44c62f52f7e8be135" exitCode=0
Sep 17 15:49:25.940040 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:25.939892    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" event={"ID":"496cd31b-be97-4723-a260-886153fc3359","Type":"ContainerDied","Data":"60fc4042f3ba0a71913dfce23850010edec356954ee41cc44c62f52f7e8be135"}
Sep 17 15:49:26.012716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.012173    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:26.012716 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.012252    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:26.039003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.038966    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:49:26.143492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143455    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143513    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gr4sg\" (UniqueName: \"kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143565    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143588    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143842 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143744    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143887 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143823    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:26.143887 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143853    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs\") pod \"496cd31b-be97-4723-a260-886153fc3359\" (UID: \"496cd31b-be97-4723-a260-886153fc3359\") "
Sep 17 15:49:26.143887 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143836    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home" (OuterVolumeSpecName: "home") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:26.144037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.143996    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache" (OuterVolumeSpecName: "model-cache") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:26.144243 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.144223    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.144317 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.144246    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.144317 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.144260    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.145789 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.145759    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm" (OuterVolumeSpecName: "dshm") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:26.145899 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.145786    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:49:26.145899 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.145785    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg" (OuterVolumeSpecName: "kube-api-access-gr4sg") pod "496cd31b-be97-4723-a260-886153fc3359" (UID: "496cd31b-be97-4723-a260-886153fc3359"). InnerVolumeSpecName "kube-api-access-gr4sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:26.245422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.245318    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496cd31b-be97-4723-a260-886153fc3359-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.245422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.245357    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4sg\" (UniqueName: \"kubernetes.io/projected/496cd31b-be97-4723-a260-886153fc3359-kube-api-access-gr4sg\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.245422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.245370    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496cd31b-be97-4723-a260-886153fc3359-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:26.633916 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.633876    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:49:26.634407 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634389    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="496cd31b-be97-4723-a260-886153fc3359" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634410    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="496cd31b-be97-4723-a260-886153fc3359" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634435    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdc4b642-4115-4c21-85fb-2967bebdbd59" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634442    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdc4b642-4115-4c21-85fb-2967bebdbd59" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634454    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634463    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634480    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c68d398-4223-4873-9532-7034966e54bc" containerName="main"
Sep 17 15:49:26.634512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634488    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c68d398-4223-4873-9532-7034966e54bc" containerName="main"
Sep 17 15:49:26.634888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634642    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="496cd31b-be97-4723-a260-886153fc3359" containerName="main"
Sep 17 15:49:26.634888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634660    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdc4b642-4115-4c21-85fb-2967bebdbd59" containerName="main"
Sep 17 15:49:26.634888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634675    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c68d398-4223-4873-9532-7034966e54bc" containerName="main"
Sep 17 15:49:26.634888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.634686    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab21152d-21df-4d71-bd8b-0da3bc0bc354" containerName="main"
Sep 17 15:49:26.638384 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.638357    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.639899 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.639871    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:49:26.640709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.640683    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-608b97-dockercfg-82654\""
Sep 17 15:49:26.641282 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.641266    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-608b97\""
Sep 17 15:49:26.649527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.649500    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.649711 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.649572    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5r48\" (UniqueName: \"kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.678178 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.678153    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:49:26.681771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.681756    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.696135 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.696112    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:49:26.750459 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.750389    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf85z\" (UniqueName: \"kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.750677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.750495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.750677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.750542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.750677 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.750575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5r48\" (UniqueName: \"kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.751524 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.751498    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.763720 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.763685    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5r48\" (UniqueName: \"kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48\") pod \"payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.852227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.852133    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf85z\" (UniqueName: \"kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.852227 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.852209    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.852956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.852936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.861329 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.861302    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf85z\" (UniqueName: \"kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z\") pod \"payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:26.948058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.948029    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"
Sep 17 15:49:26.948256 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.948067    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8" event={"ID":"496cd31b-be97-4723-a260-886153fc3359","Type":"ContainerDied","Data":"1bb2a97774b9edfb6f9b5df7d88b14f3e6ab628c7a0aae902df2b491a1487493"}
Sep 17 15:49:26.948256 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.948119    2564 scope.go:117] "RemoveContainer" containerID="60fc4042f3ba0a71913dfce23850010edec356954ee41cc44c62f52f7e8be135"
Sep 17 15:49:26.951836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.951811    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:26.966987 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.966958    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:49:26.969584 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.969564    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/e2e-conflict-a-bd4c6f71-kserve-844c7bbc7b-bskw8"]
Sep 17 15:49:26.991775 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:26.991743    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:27.024694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.024156    2564 patch_prober.go:28] interesting pod/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body=
Sep 17 15:49:27.024694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.024235    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused"
Sep 17 15:49:27.254161 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.254131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:49:27.255707 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:49:27.255680    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfd4851_b23f_4d47_ae00_bda776ff88c0.slice/crio-3f2a8a16274645f576954b29cd96642a8e039981e8e85be3f4d66c3e9a80694d WatchSource:0}: Error finding container 3f2a8a16274645f576954b29cd96642a8e039981e8e85be3f4d66c3e9a80694d: Status 404 returned error can't find the container with id 3f2a8a16274645f576954b29cd96642a8e039981e8e85be3f4d66c3e9a80694d
Sep 17 15:49:27.313858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.313829    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:49:27.337868 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:49:27.337837    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057447e2_1ea6_413d_b791_6a2e43ead46b.slice/crio-73ef774cad74347f78eb5f9cf67142e49cec13764f2876ea6e920cf56d4188bb WatchSource:0}: Error finding container 73ef774cad74347f78eb5f9cf67142e49cec13764f2876ea6e920cf56d4188bb: Status 404 returned error can't find the container with id 73ef774cad74347f78eb5f9cf67142e49cec13764f2876ea6e920cf56d4188bb
Sep 17 15:49:27.957066 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.957028    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" event={"ID":"057447e2-1ea6-413d-b791-6a2e43ead46b","Type":"ContainerStarted","Data":"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"}
Sep 17 15:49:27.957278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.957099    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:27.957278 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.957126    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" event={"ID":"057447e2-1ea6-413d-b791-6a2e43ead46b","Type":"ContainerStarted","Data":"73ef774cad74347f78eb5f9cf67142e49cec13764f2876ea6e920cf56d4188bb"}
Sep 17 15:49:27.959577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.959545    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" event={"ID":"5dfd4851-b23f-4d47-ae00-bda776ff88c0","Type":"ContainerStarted","Data":"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"}
Sep 17 15:49:27.959724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.959598    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" event={"ID":"5dfd4851-b23f-4d47-ae00-bda776ff88c0","Type":"ContainerStarted","Data":"3f2a8a16274645f576954b29cd96642a8e039981e8e85be3f4d66c3e9a80694d"}
Sep 17 15:49:27.959785 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.959743    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:27.975537 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.975497    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" podStartSLOduration=1.975484351 podStartE2EDuration="1.975484351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:27.973789631 +0000 UTC m=+1159.940967475" watchObservedRunningTime="2026-09-17 15:49:27.975484351 +0000 UTC m=+1159.942662169"
Sep 17 15:49:27.989938 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:27.989898    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" podStartSLOduration=1.9898800269999999 podStartE2EDuration="1.989880027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:27.989074015 +0000 UTC m=+1159.956251856" watchObservedRunningTime="2026-09-17 15:49:27.989880027 +0000 UTC m=+1159.957057845"
Sep 17 15:49:28.015323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:28.015300    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:49:28.016375 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:28.016355    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:49:28.536509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:28.536470    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="496cd31b-be97-4723-a260-886153fc3359" path="/var/lib/kubelet/pods/496cd31b-be97-4723-a260-886153fc3359/volumes"
Sep 17 15:49:31.544186 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.544104    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:49:31.548552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.548531    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.551060 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.551032    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"kube-root-ca.crt\""
Sep 17 15:49:31.551718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.551692    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"distinct-w4-c86e9a-kserve-self-signed-certs\""
Sep 17 15:49:31.551718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.551711    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"openshift-service-ca.crt\""
Sep 17 15:49:31.551858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.551705    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c86e9a\"/\"default-dockercfg-287z2\""
Sep 17 15:49:31.556957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.556857    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:49:31.600976 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.600943    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.600976 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.600978    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.601210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.601027    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.601210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.601056    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.601210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.601102    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.601210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.601148    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbw9\" (UniqueName: \"kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701675 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701634    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701675 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701752    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbw9\" (UniqueName: \"kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701808    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.701948 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.701827    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.702173 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.702106    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.702173 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.702141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.702294 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.702184    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.703904 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.703876    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.704248 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.704231    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.710235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.710210    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbw9\" (UniqueName: \"kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9\") pod \"distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") " pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:31.859497 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:31.859407    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:32.137193 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:32.137170    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:49:32.139019 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:49:32.138993    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22c95a57_a6c1_46d3_baab_87c4b01ff856.slice/crio-7516ca5e9b00998d2b7220f8e556b476bfc6e2b3792a12b0e7ee5e68ebea14c1 WatchSource:0}: Error finding container 7516ca5e9b00998d2b7220f8e556b476bfc6e2b3792a12b0e7ee5e68ebea14c1: Status 404 returned error can't find the container with id 7516ca5e9b00998d2b7220f8e556b476bfc6e2b3792a12b0e7ee5e68ebea14c1
Sep 17 15:49:32.977866 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:32.977831    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:49:32.991282 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:32.991247    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" event={"ID":"22c95a57-a6c1-46d3-baab-87c4b01ff856","Type":"ContainerStarted","Data":"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"}
Sep 17 15:49:32.991430 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:32.991290    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" event={"ID":"22c95a57-a6c1-46d3-baab-87c4b01ff856","Type":"ContainerStarted","Data":"7516ca5e9b00998d2b7220f8e556b476bfc6e2b3792a12b0e7ee5e68ebea14c1"}
Sep 17 15:49:33.012794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:33.012734    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" podStartSLOduration=2.012715417 podStartE2EDuration="2.012715417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:33.008964553 +0000 UTC m=+1164.976142400" watchObservedRunningTime="2026-09-17 15:49:33.012715417 +0000 UTC m=+1164.979893236"
Sep 17 15:49:37.956466 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:37.956426    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:49:37.956914 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:37.956682    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy" containerID="cri-o://fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4" gracePeriod=30
Sep 17 15:49:38.330073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.330042    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"]
Sep 17 15:49:38.333671 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.333648    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:38.337137 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.337115    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:49:38.337251 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.337154    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:49:38.347479 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.347455    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"]
Sep 17 15:49:38.471366 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.471328    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d\" (UID: \"a78a701d-4158-41c2-b281-f2b85ebc6f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:38.572954 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.572916    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d\" (UID: \"a78a701d-4158-41c2-b281-f2b85ebc6f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:38.573551 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.573531    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d\" (UID: \"a78a701d-4158-41c2-b281-f2b85ebc6f85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:38.644860 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.644762    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:38.925008 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.924972    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"]
Sep 17 15:49:38.926352 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:49:38.926326    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda78a701d_4158_41c2_b281_f2b85ebc6f85.slice/crio-97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969 WatchSource:0}: Error finding container 97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969: Status 404 returned error can't find the container with id 97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969
Sep 17 15:49:38.968545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.968515    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:49:38.968879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:38.968585    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:49:39.029683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:39.029648    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d" event={"ID":"a78a701d-4158-41c2-b281-f2b85ebc6f85","Type":"ContainerStarted","Data":"97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969"}
Sep 17 15:49:40.036905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:40.036869    2564 generic.go:356] "Generic (PLEG): container finished" podID="a78a701d-4158-41c2-b281-f2b85ebc6f85" containerID="7f6498e0987868e9a807a1cea38f3b6dc5c0425b7c9dfbb5b4f8ba54a281286f" exitCode=0
Sep 17 15:49:40.037309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:40.036928    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d" event={"ID":"a78a701d-4158-41c2-b281-f2b85ebc6f85","Type":"ContainerDied","Data":"7f6498e0987868e9a807a1cea38f3b6dc5c0425b7c9dfbb5b4f8ba54a281286f"}
Sep 17 15:49:41.180794 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.180769    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:41.300595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.300509    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca\") pod \"a78a701d-4158-41c2-b281-f2b85ebc6f85\" (UID: \"a78a701d-4158-41c2-b281-f2b85ebc6f85\") "
Sep 17 15:49:41.300894 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.300873    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a78a701d-4158-41c2-b281-f2b85ebc6f85" (UID: "a78a701d-4158-41c2-b281-f2b85ebc6f85"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:49:41.402275 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.402243    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a78a701d-4158-41c2-b281-f2b85ebc6f85-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:41.859938 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.859895    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:41.860145 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.860083    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:41.873545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:41.873517    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:42.051216 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:42.051190    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"
Sep 17 15:49:42.051382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:42.051223    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d" event={"ID":"a78a701d-4158-41c2-b281-f2b85ebc6f85","Type":"ContainerDied","Data":"97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969"}
Sep 17 15:49:42.051382 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:42.051274    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97bf8b7838be3827395ccaf471502de78d9ce2684302795ce55a4bfe654e4969"
Sep 17 15:49:42.064698 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:42.064667    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:49:43.235541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.235484    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:49:43.322752 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322716    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.322919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322778    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.322919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322805    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.322919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.322919 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.323189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322942    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.323189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.322969    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.323189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323007    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:43.323189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323022    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.323189 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323054    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5pd2\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2\") pod \"735434a5-d970-4b74-88ec-c6cd5148a1d0\" (UID: \"735434a5-d970-4b74-88ec-c6cd5148a1d0\") "
Sep 17 15:49:43.323601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323202    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:43.323601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323338    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:49:43.323601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323469    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:43.323601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323495    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:43.323820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323633    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.323820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323654    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.323820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323669    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.323820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323682    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/735434a5-d970-4b74-88ec-c6cd5148a1d0-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.323820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.323693    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.325211 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.325184    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:43.325285 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.325215    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2" (OuterVolumeSpecName: "kube-api-access-x5pd2") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "kube-api-access-x5pd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:43.325496 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.325471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:43.325496 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.325471    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "735434a5-d970-4b74-88ec-c6cd5148a1d0" (UID: "735434a5-d970-4b74-88ec-c6cd5148a1d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:49:43.424488 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.424458    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5pd2\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-kube-api-access-x5pd2\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.424488 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.424485    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.424676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.424495    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:43.424676 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:43.424503    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/735434a5-d970-4b74-88ec-c6cd5148a1d0-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:49:44.064609 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.064574    2564 generic.go:356] "Generic (PLEG): container finished" podID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerID="fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4" exitCode=0
Sep 17 15:49:44.064793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.064648    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"
Sep 17 15:49:44.064793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.064672    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerDied","Data":"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"}
Sep 17 15:49:44.064793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.064732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr" event={"ID":"735434a5-d970-4b74-88ec-c6cd5148a1d0","Type":"ContainerDied","Data":"a2872c613c3a99bace14049fecf1d6fa86222aa9472fb008e7397993d665a205"}
Sep 17 15:49:44.064793 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.064757    2564 scope.go:117] "RemoveContainer" containerID="fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"
Sep 17 15:49:44.078388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.078156    2564 scope.go:117] "RemoveContainer" containerID="86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"
Sep 17 15:49:44.086435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.086419    2564 scope.go:117] "RemoveContainer" containerID="fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"
Sep 17 15:49:44.086703 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:49:44.086681    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4\": container with ID starting with fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4 not found: ID does not exist" containerID="fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"
Sep 17 15:49:44.086761 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.086711    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4"} err="failed to get container status \"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4\": rpc error: code = NotFound desc = could not find container \"fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4\": container with ID starting with fdbc656706b5ed2dd2fe550662367129b70b15e6caa12d25de0af8cd802566e4 not found: ID does not exist"
Sep 17 15:49:44.086761 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.086728    2564 scope.go:117] "RemoveContainer" containerID="86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"
Sep 17 15:49:44.086979 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:49:44.086958    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94\": container with ID starting with 86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94 not found: ID does not exist" containerID="86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"
Sep 17 15:49:44.087033 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.086989    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94"} err="failed to get container status \"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94\": rpc error: code = NotFound desc = could not find container \"86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94\": container with ID starting with 86ed41abdaee3a499dc881581b790ab90c6290e961aa8e5b92a7571aef609c94 not found: ID does not exist"
Sep 17 15:49:44.091643 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.091609    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:49:44.095543 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.095525    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-449c47-openshift-default-6f85596644-gwlzr"]
Sep 17 15:49:44.525868 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:44.525835    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" path="/var/lib/kubelet/pods/735434a5-d970-4b74-88ec-c6cd5148a1d0/volumes"
Sep 17 15:49:50.017689 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:50.017601    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"]
Sep 17 15:49:50.019690 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:50.019667    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a2dc23-lwt9d"]
Sep 17 15:49:50.526336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:50.526306    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a78a701d-4158-41c2-b281-f2b85ebc6f85" path="/var/lib/kubelet/pods/a78a701d-4158-41c2-b281-f2b85ebc6f85/volumes"
Sep 17 15:49:55.607652 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:55.606036    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:49:57.372570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:57.372533    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:49:57.373040 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:49:57.372832    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy" containerID="cri-o://aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562" gracePeriod=30
Sep 17 15:50:02.185332 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185304    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:50:02.185775 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185760    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a78a701d-4158-41c2-b281-f2b85ebc6f85" containerName="revoke-keys"
Sep 17 15:50:02.185846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185777    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a78a701d-4158-41c2-b281-f2b85ebc6f85" containerName="revoke-keys"
Sep 17 15:50:02.185846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185790    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.185846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185795    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.185846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185801    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.185846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185807    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.186014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185882    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a78a701d-4158-41c2-b281-f2b85ebc6f85" containerName="revoke-keys"
Sep 17 15:50:02.186014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185893    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.186014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.185900    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="735434a5-d970-4b74-88ec-c6cd5148a1d0" containerName="istio-proxy"
Sep 17 15:50:02.190595 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.190579    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.192950 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.192932    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-99719fde-dockercfg-w7ck2\""
Sep 17 15:50:02.193337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.193316    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-99719fde\""
Sep 17 15:50:02.200320 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.200301    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:50:02.220325 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.220301    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.220435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.220397    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5swr\" (UniqueName: \"kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.269148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.269119    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:50:02.274687 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.274488    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.288789 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.288757    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:50:02.328937 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.328900    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.329402 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.329368    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5swr\" (UniqueName: \"kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.330384 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.330357    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.338071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.338047    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5swr\" (UniqueName: \"kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr\") pod \"payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.430738 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.430714    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.430839 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.430779    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd2s2\" (UniqueName: \"kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.501291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.501263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:02.532210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.532176    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.532662 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.532326    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd2s2\" (UniqueName: \"kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.532971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.532949    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.544636 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.544592    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd2s2\" (UniqueName: \"kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2\") pod \"payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") " pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.591087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.591046    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:02.632427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.632400    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:50:02.632911 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.632890    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.632993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.632928    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.632993 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.632978    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633078 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633068    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633134 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633095    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633137    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj56n\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633144    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:02.633190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633172    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633346 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633221    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633346 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633270    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token\") pod \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\" (UID: \"0d1fbbd0-b275-4843-ae48-8978d1e2fca1\") "
Sep 17 15:50:02.633346 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633248    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:02.633606 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633420    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:50:02.633739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633703    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:02.633739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633709    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.633739 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633738    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.633897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633749    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.633897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.633845    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:02.635455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.635420    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n" (OuterVolumeSpecName: "kube-api-access-kj56n") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "kube-api-access-kj56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:50:02.635859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.635831    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:50:02.635949 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.635836    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:02.635996 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.635984    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d1fbbd0-b275-4843-ae48-8978d1e2fca1" (UID: "0d1fbbd0-b275-4843-ae48-8978d1e2fca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:50:02.734500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734466    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj56n\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-kube-api-access-kj56n\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.734500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734496    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.734500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734506    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.734765 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734514    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.734765 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734522    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.734765 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.734530    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d1fbbd0-b275-4843-ae48-8978d1e2fca1-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:50:02.788927 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.788899    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:50:02.790786 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:02.790758    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e3eb54_331a_4b4b_a29e_1250b0ad4349.slice/crio-dd6150cb94e1ee7eefc345c58ddaa21f24e0076c2b7e37d25b36e79900bfee80 WatchSource:0}: Error finding container dd6150cb94e1ee7eefc345c58ddaa21f24e0076c2b7e37d25b36e79900bfee80: Status 404 returned error can't find the container with id dd6150cb94e1ee7eefc345c58ddaa21f24e0076c2b7e37d25b36e79900bfee80
Sep 17 15:50:02.874021 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:02.873999    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:50:02.876016 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:02.875990    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf96195b_20f9_4bf4_80f6_6f9f06acc54d.slice/crio-25d050e4f0b3bb9006e489b204829c69dadfaaedd01b73655630ddcca42beede WatchSource:0}: Error finding container 25d050e4f0b3bb9006e489b204829c69dadfaaedd01b73655630ddcca42beede: Status 404 returned error can't find the container with id 25d050e4f0b3bb9006e489b204829c69dadfaaedd01b73655630ddcca42beede
Sep 17 15:50:03.183361 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.183323    2564 generic.go:356] "Generic (PLEG): container finished" podID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerID="aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562" exitCode=0
Sep 17 15:50:03.183553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.183374    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"
Sep 17 15:50:03.183553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.183406    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerDied","Data":"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"}
Sep 17 15:50:03.183553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.183458    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx" event={"ID":"0d1fbbd0-b275-4843-ae48-8978d1e2fca1","Type":"ContainerDied","Data":"1953e75ce757f60184497447f88733eb21f1d4bdec64e6d5aadb8c48a8b79bf9"}
Sep 17 15:50:03.183553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.183489    2564 scope.go:117] "RemoveContainer" containerID="aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"
Sep 17 15:50:03.185450 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.185412    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" event={"ID":"d1e3eb54-331a-4b4b-a29e-1250b0ad4349","Type":"ContainerStarted","Data":"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"}
Sep 17 15:50:03.185935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.185467    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" event={"ID":"d1e3eb54-331a-4b4b-a29e-1250b0ad4349","Type":"ContainerStarted","Data":"dd6150cb94e1ee7eefc345c58ddaa21f24e0076c2b7e37d25b36e79900bfee80"}
Sep 17 15:50:03.185935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.185500    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:03.187452 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.187426    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" event={"ID":"cf96195b-20f9-4bf4-80f6-6f9f06acc54d","Type":"ContainerStarted","Data":"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"}
Sep 17 15:50:03.187577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.187476    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" event={"ID":"cf96195b-20f9-4bf4-80f6-6f9f06acc54d","Type":"ContainerStarted","Data":"25d050e4f0b3bb9006e489b204829c69dadfaaedd01b73655630ddcca42beede"}
Sep 17 15:50:03.187577 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.187508    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:03.193890 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.193874    2564 scope.go:117] "RemoveContainer" containerID="3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"
Sep 17 15:50:03.202362 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.202342    2564 scope.go:117] "RemoveContainer" containerID="aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"
Sep 17 15:50:03.202639 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:50:03.202602    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562\": container with ID starting with aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562 not found: ID does not exist" containerID="aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"
Sep 17 15:50:03.202705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.202644    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562"} err="failed to get container status \"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562\": rpc error: code = NotFound desc = could not find container \"aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562\": container with ID starting with aed385119c8a85fde5a133d7ec9264e9ad5bf5f2310321f1675d890e33e9b562 not found: ID does not exist"
Sep 17 15:50:03.202705 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.202663    2564 scope.go:117] "RemoveContainer" containerID="3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"
Sep 17 15:50:03.202898 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:50:03.202884    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b\": container with ID starting with 3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b not found: ID does not exist" containerID="3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"
Sep 17 15:50:03.202941 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.202901    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b"} err="failed to get container status \"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b\": rpc error: code = NotFound desc = could not find container \"3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b\": container with ID starting with 3725b59c4fc1102c5a86bb6e4839f0bd42001c466fe49410626f4f6288bb0a7b not found: ID does not exist"
Sep 17 15:50:03.205560 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.205483    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" podStartSLOduration=1.205464094 podStartE2EDuration="1.205464094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:03.204719229 +0000 UTC m=+1195.171897066" watchObservedRunningTime="2026-09-17 15:50:03.205464094 +0000 UTC m=+1195.172641917"
Sep 17 15:50:03.229816 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.229778    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" podStartSLOduration=1.229767254 podStartE2EDuration="1.229767254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:03.225728302 +0000 UTC m=+1195.192906121" watchObservedRunningTime="2026-09-17 15:50:03.229767254 +0000 UTC m=+1195.196945072"
Sep 17 15:50:03.246094 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.246073    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:50:03.248770 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.248751    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a2dc23-openshift-default-57d9c95d8b-hqppx"]
Sep 17 15:50:03.446039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.445960    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:50:03.446388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.446375    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:03.446388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.446388    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:03.446500 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.446458    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:03.450870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.450852    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.455869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.455849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"kube-root-ca.crt\""
Sep 17 15:50:03.455869 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.455866    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"default-dockercfg-k96cw\""
Sep 17 15:50:03.456029 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.455849    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"openshift-service-ca.crt\""
Sep 17 15:50:03.456202 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.456185    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"distinct-2-w0-849b71-kserve-self-signed-certs\""
Sep 17 15:50:03.470526 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.470505    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:50:03.545110 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545076    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.545250 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545122    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.545295 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545244    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h48bl\" (UniqueName: \"kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.545295 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545272    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.545461 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545299    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.545461 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.545322    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646662    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h48bl\" (UniqueName: \"kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646708    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646734    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646756    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646861    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.646966 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.646887    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.647157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.647121    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.647214 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.647180    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.647272 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.647226    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.649194 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.649171    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.649309 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.649289    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.661206 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.661177    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h48bl\" (UniqueName: \"kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl\") pod \"distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") " pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:03.760999 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:03.760970    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:04.046909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:04.046875    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:50:04.047928 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:04.047904    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd88a93c1_9fe5_4579_a09a_60b2ad14fbe3.slice/crio-c1346165375b77242cdaa8917720055bc5d80fbb5bb08a3f9c85e688a422ea90 WatchSource:0}: Error finding container c1346165375b77242cdaa8917720055bc5d80fbb5bb08a3f9c85e688a422ea90: Status 404 returned error can't find the container with id c1346165375b77242cdaa8917720055bc5d80fbb5bb08a3f9c85e688a422ea90
Sep 17 15:50:04.196084 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:04.196049    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" event={"ID":"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3","Type":"ContainerStarted","Data":"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"}
Sep 17 15:50:04.196455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:04.196092    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" event={"ID":"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3","Type":"ContainerStarted","Data":"c1346165375b77242cdaa8917720055bc5d80fbb5bb08a3f9c85e688a422ea90"}
Sep 17 15:50:04.222031 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:04.221978    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" podStartSLOduration=1.221959158 podStartE2EDuration="1.221959158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:04.216854541 +0000 UTC m=+1196.184032360" watchObservedRunningTime="2026-09-17 15:50:04.221959158 +0000 UTC m=+1196.189136978"
Sep 17 15:50:04.531408 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:04.531366    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" path="/var/lib/kubelet/pods/0d1fbbd0-b275-4843-ae48-8978d1e2fca1/volumes"
Sep 17 15:50:08.721698 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:08.721667    2564 scope.go:117] "RemoveContainer" containerID="919f3ebf8dc75b9e4f26e507ae5d3e14a401cf7f40905570edb257a1186c272a"
Sep 17 15:50:13.761261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:13.761223    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:13.761744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:13.761276    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:13.774100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:13.774073    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:14.198559 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:14.198474    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:50:14.198724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:14.198676    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:50:14.270873 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:14.270842    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:50:27.199127 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:27.199089    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:50:32.718913 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.718880    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 15:50:32.719371 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.719356    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:32.719420 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.719372    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:32.719475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.719465    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d1fbbd0-b275-4843-ae48-8978d1e2fca1" containerName="istio-proxy"
Sep 17 15:50:32.724458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.724439    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.727933 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.727913    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-5818c9-openshift-default-dockercfg-z6wqd\""
Sep 17 15:50:32.743894 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.743868    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 15:50:32.838126 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838093    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7r65\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838131    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838238    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838302 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838273    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838304    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838346    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838414 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838385    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838441    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.838512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.838484    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939458 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939425    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939484    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7r65\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939510    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939596    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939654    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939758    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939788    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939911    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.939956 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939932    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.940210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.939964    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.940210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.940123    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.940210 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.940169    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.940385 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.940291    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.942052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.942025    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.942317 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.942301    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.951453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.951434    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:32.951548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:32.951459    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7r65\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65\") pod \"e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") " pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:33.034905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.034869    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:33.333071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.333044    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 15:50:33.334466 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:33.334433    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod935c4812_9d5e_4d83_b314_aaedd56b9936.slice/crio-9575e57983242626122cd673f9a1cb92c47126430800d16ecfa3f2ee3f3d40dc WatchSource:0}: Error finding container 9575e57983242626122cd673f9a1cb92c47126430800d16ecfa3f2ee3f3d40dc: Status 404 returned error can't find the container with id 9575e57983242626122cd673f9a1cb92c47126430800d16ecfa3f2ee3f3d40dc
Sep 17 15:50:33.336957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.336898    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:50:33.337041 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.336993    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:50:33.337041 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.337022    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:50:33.377846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:33.377815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerStarted","Data":"9575e57983242626122cd673f9a1cb92c47126430800d16ecfa3f2ee3f3d40dc"}
Sep 17 15:50:34.386276 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:34.386234    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerStarted","Data":"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"}
Sep 17 15:50:34.426426 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:34.426365    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podStartSLOduration=2.426342988 podStartE2EDuration="2.426342988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:34.425798616 +0000 UTC m=+1226.392976435" watchObservedRunningTime="2026-09-17 15:50:34.426342988 +0000 UTC m=+1226.393520808"
Sep 17 15:50:35.035282 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:35.035252    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:35.040432 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:35.040409    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:35.392140 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:35.392061    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:35.393160 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:35.393140    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:50:43.666014 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.665971    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 15:50:43.670052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.670030    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.672754 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.672731    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-5818c9-dockercfg-8qwnx\""
Sep 17 15:50:43.672870 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.672850    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-5818c9\""
Sep 17 15:50:43.687638 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.687602    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 15:50:43.751039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.750167    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.751039 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.750256    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhw5\" (UniqueName: \"kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.851378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.851343    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhw5\" (UniqueName: \"kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.851568 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.851485    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.852225 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.852203    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.860582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.860556    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhw5\" (UniqueName: \"kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5\") pod \"payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:43.981112 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:43.981037    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:44.260336 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:44.260312    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 15:50:44.261581 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:44.261555    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod591c1e24_e9d0_4ea8_922c_615548f9db80.slice/crio-038bc181e12e3a8cfd861e6cfacb8eec75bcbe91cd01aa9c9ee6882fcc3a754d WatchSource:0}: Error finding container 038bc181e12e3a8cfd861e6cfacb8eec75bcbe91cd01aa9c9ee6882fcc3a754d: Status 404 returned error can't find the container with id 038bc181e12e3a8cfd861e6cfacb8eec75bcbe91cd01aa9c9ee6882fcc3a754d
Sep 17 15:50:44.450921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:44.450884    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" event={"ID":"591c1e24-e9d0-4ea8-922c-615548f9db80","Type":"ContainerStarted","Data":"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"}
Sep 17 15:50:44.451105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:44.450939    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" event={"ID":"591c1e24-e9d0-4ea8-922c-615548f9db80","Type":"ContainerStarted","Data":"038bc181e12e3a8cfd861e6cfacb8eec75bcbe91cd01aa9c9ee6882fcc3a754d"}
Sep 17 15:50:44.451105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:44.450962    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:44.484264 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:44.484218    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" podStartSLOduration=1.48420295 podStartE2EDuration="1.48420295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:44.482882001 +0000 UTC m=+1236.450059819" watchObservedRunningTime="2026-09-17 15:50:44.48420295 +0000 UTC m=+1236.451380796"
Sep 17 15:50:54.287100 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.287069    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:50:54.291029 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.291007    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.295338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.295316    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-99719fde\"/\"kube-root-ca.crt\""
Sep 17 15:50:54.295338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.295332    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-99719fde\"/\"default-dockercfg-knw6t\""
Sep 17 15:50:54.295525 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.295343    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-99719fde\"/\"openshift-service-ca.crt\""
Sep 17 15:50:54.295635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.295600    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-99719fde\"/\"e2e-lifecycle-model-99719fde-kserve-self-signed-certs\""
Sep 17 15:50:54.312129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.312108    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:50:54.351916 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.351891    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjvzc\" (UniqueName: \"kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.352044 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.351947    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.352092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.352051    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.352138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.352092    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.352138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.352133    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.352220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.352173    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453372 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453339    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453384    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjvzc\" (UniqueName: \"kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453548 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453422    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453687 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453582    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453745 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453691    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453796 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453747    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.453940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.453923    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.454092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.454059    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.454092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.454076    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.455642 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.455609    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.455903 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.455888    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.464570 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.464550    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjvzc\" (UniqueName: \"kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc\") pod \"e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") " pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.601040 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.600962    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:50:54.877132 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:54.877109    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:50:54.879357 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:50:54.879329    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55cc9579_f330_466e_9e3f_705156525b0d.slice/crio-d03d2693f572e331daa7414163ff518923bcdd84107a96e7d62d8f1b9aef32ca WatchSource:0}: Error finding container d03d2693f572e331daa7414163ff518923bcdd84107a96e7d62d8f1b9aef32ca: Status 404 returned error can't find the container with id d03d2693f572e331daa7414163ff518923bcdd84107a96e7d62d8f1b9aef32ca
Sep 17 15:50:55.458593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:55.458563    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 15:50:55.522518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:55.522487    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" event={"ID":"55cc9579-f330-466e-9e3f-705156525b0d","Type":"ContainerStarted","Data":"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"}
Sep 17 15:50:55.522518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:55.522522    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" event={"ID":"55cc9579-f330-466e-9e3f-705156525b0d","Type":"ContainerStarted","Data":"d03d2693f572e331daa7414163ff518923bcdd84107a96e7d62d8f1b9aef32ca"}
Sep 17 15:50:55.543473 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:55.543423    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" podStartSLOduration=1.5434041330000001 podStartE2EDuration="1.543404133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:55.542876467 +0000 UTC m=+1247.510054283" watchObservedRunningTime="2026-09-17 15:50:55.543404133 +0000 UTC m=+1247.510581952"
Sep 17 15:50:58.176820 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:50:58.176788    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:51:04.601240 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:04.601156    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:51:04.601240 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:04.601213    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:51:04.638706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:04.638673    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:51:05.593256 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:05.593168    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:51:06.433809 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:06.433765    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 15:51:06.434281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:06.433850    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 15:51:08.601138 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.601106    2564 generic.go:356] "Generic (PLEG): container finished" podID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerID="9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf" exitCode=0
Sep 17 15:51:08.601543 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.601184    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerDied","Data":"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"}
Sep 17 15:51:08.601675 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.601660    2564 scope.go:117] "RemoveContainer" containerID="9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"
Sep 17 15:51:08.603015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.602986    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:51:08.603090 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.603057    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:51:08.603128 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.603088    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:51:08.777323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.777294    2564 scope.go:117] "RemoveContainer" containerID="2cf738ad3f041f2b6bc0b0b5448174b02398bd67b76e70a6db69de037cc688ab"
Sep 17 15:51:08.787505 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.787483    2564 scope.go:117] "RemoveContainer" containerID="4dc5397100d33e99952f708cf77d0b7a834d7e6a446adaca3ba7f009a5d58d37"
Sep 17 15:51:08.800516 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:08.800492    2564 scope.go:117] "RemoveContainer" containerID="b5ae3bb3b61f0305b440c7ccc52c9a3fbe60fd3db0b59b78d856eaec114155fa"
Sep 17 15:51:09.614069 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:09.614026    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerStarted","Data":"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"}
Sep 17 15:51:10.035173 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:10.035134    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:51:10.035381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:10.035357    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 15:51:10.035438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:10.035411    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 15:51:11.035970 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:11.035931    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 15:51:11.036417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:11.035990    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 15:51:12.035338 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:12.035294    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 15:51:12.035536 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:12.035362    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 15:51:13.035239 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:13.035196    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 15:51:13.035752 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:13.035270    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 15:51:14.036099 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:14.036054    2564 patch_prober.go:28] interesting pod/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body=
Sep 17 15:51:14.036587 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:14.036142    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused"
Sep 17 15:51:15.039002 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:15.038963    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:51:15.039396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:15.039226    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:51:15.039828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:15.039806    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 15:51:35.318172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:51:35.318141    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:52:04.756188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.756153    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"]
Sep 17 15:52:04.758880 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.758857    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:04.763940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.763911    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:52:04.764082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.764052    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:52:04.772474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.772449    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"]
Sep 17 15:52:04.897032 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.896997    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99719fde-6nb7j\" (UID: \"eabf247a-14ee-4356-b137-68b291137c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:04.997608 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.997575    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99719fde-6nb7j\" (UID: \"eabf247a-14ee-4356-b137-68b291137c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:04.998241 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:04.998224    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99719fde-6nb7j\" (UID: \"eabf247a-14ee-4356-b137-68b291137c51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:05.069354 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:05.069319    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:05.342671 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:05.342646    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"]
Sep 17 15:52:05.344445 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:52:05.344420    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabf247a_14ee_4356_b137_68b291137c51.slice/crio-3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2 WatchSource:0}: Error finding container 3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2: Status 404 returned error can't find the container with id 3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2
Sep 17 15:52:05.982762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:05.982677    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j" event={"ID":"eabf247a-14ee-4356-b137-68b291137c51","Type":"ContainerStarted","Data":"0ed459c72847adc72c5af23c3102de9a94f7bf68e4167616f85a3c5089ac5a57"}
Sep 17 15:52:05.982762 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:05.982736    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j" event={"ID":"eabf247a-14ee-4356-b137-68b291137c51","Type":"ContainerStarted","Data":"3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2"}
Sep 17 15:52:06.002787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:06.002731    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j" podStartSLOduration=2.002713507 podStartE2EDuration="2.002713507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:06.001279903 +0000 UTC m=+1317.968457747" watchObservedRunningTime="2026-09-17 15:52:06.002713507 +0000 UTC m=+1317.969891324"
Sep 17 15:52:06.989645 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:06.989598    2564 generic.go:356] "Generic (PLEG): container finished" podID="eabf247a-14ee-4356-b137-68b291137c51" containerID="0ed459c72847adc72c5af23c3102de9a94f7bf68e4167616f85a3c5089ac5a57" exitCode=0
Sep 17 15:52:06.990033 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:06.989645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j" event={"ID":"eabf247a-14ee-4356-b137-68b291137c51","Type":"ContainerDied","Data":"0ed459c72847adc72c5af23c3102de9a94f7bf68e4167616f85a3c5089ac5a57"}
Sep 17 15:52:08.142929 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:08.142903    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:08.230271 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:08.230239    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca\") pod \"eabf247a-14ee-4356-b137-68b291137c51\" (UID: \"eabf247a-14ee-4356-b137-68b291137c51\") "
Sep 17 15:52:08.230594 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:08.230572    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eabf247a-14ee-4356-b137-68b291137c51" (UID: "eabf247a-14ee-4356-b137-68b291137c51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:08.332022 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:08.331985    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabf247a-14ee-4356-b137-68b291137c51-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:08.833953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:08.833924    2564 scope.go:117] "RemoveContainer" containerID="c9c11e2fd3d48d207891bea9fa1dff0de9b0afd4207ede87ccfd9d48a8ca1da1"
Sep 17 15:52:09.002443 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:09.002417    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"
Sep 17 15:52:09.002594 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:09.002455    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j" event={"ID":"eabf247a-14ee-4356-b137-68b291137c51","Type":"ContainerDied","Data":"3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2"}
Sep 17 15:52:09.002594 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:09.002537    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3de73802e54fc297ef389a88546d680cf821c446016c0bd0c607944d7fd22de2"
Sep 17 15:52:12.111199 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:12.111161    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:52:20.551424 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.551392    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:52:20.551926 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.551709    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" podUID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" containerName="payload-processing" containerID="cri-o://d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5" gracePeriod=30
Sep 17 15:52:20.565778 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.565593    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:52:20.566502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.566421    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" podUID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" containerName="payload-pre-processing" containerID="cri-o://a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435" gracePeriod=30
Sep 17 15:52:20.810791 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.810695    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"]
Sep 17 15:52:20.815554 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.815524    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99719fde-6nb7j"]
Sep 17 15:52:20.847682 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.847655    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:52:20.889593 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.889556    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:52:20.963823 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.963786    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd2s2\" (UniqueName: \"kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2\") pod \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") "
Sep 17 15:52:20.963984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.963857    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5swr\" (UniqueName: \"kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr\") pod \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") "
Sep 17 15:52:20.963984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.963906    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume\") pod \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\" (UID: \"d1e3eb54-331a-4b4b-a29e-1250b0ad4349\") "
Sep 17 15:52:20.963984 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.963965    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume\") pod \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\" (UID: \"cf96195b-20f9-4bf4-80f6-6f9f06acc54d\") "
Sep 17 15:52:20.964364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.964333    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d1e3eb54-331a-4b4b-a29e-1250b0ad4349" (UID: "d1e3eb54-331a-4b4b-a29e-1250b0ad4349"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:20.964364 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.964355    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf96195b-20f9-4bf4-80f6-6f9f06acc54d" (UID: "cf96195b-20f9-4bf4-80f6-6f9f06acc54d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:20.965980 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.965954    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2" (OuterVolumeSpecName: "kube-api-access-gd2s2") pod "cf96195b-20f9-4bf4-80f6-6f9f06acc54d" (UID: "cf96195b-20f9-4bf4-80f6-6f9f06acc54d"). InnerVolumeSpecName "kube-api-access-gd2s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:20.966081 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:20.966020    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr" (OuterVolumeSpecName: "kube-api-access-n5swr") pod "d1e3eb54-331a-4b4b-a29e-1250b0ad4349" (UID: "d1e3eb54-331a-4b4b-a29e-1250b0ad4349"). InnerVolumeSpecName "kube-api-access-n5swr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:21.065261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.065181    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd2s2\" (UniqueName: \"kubernetes.io/projected/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-kube-api-access-gd2s2\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:21.065261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.065211    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5swr\" (UniqueName: \"kubernetes.io/projected/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-kube-api-access-n5swr\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:21.065261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.065222    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1e3eb54-331a-4b4b-a29e-1250b0ad4349-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:21.065261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.065230    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf96195b-20f9-4bf4-80f6-6f9f06acc54d-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:21.073583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.073553    2564 generic.go:356] "Generic (PLEG): container finished" podID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" containerID="d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5" exitCode=0
Sep 17 15:52:21.073774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.073635    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"
Sep 17 15:52:21.073774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.073645    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" event={"ID":"cf96195b-20f9-4bf4-80f6-6f9f06acc54d","Type":"ContainerDied","Data":"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"}
Sep 17 15:52:21.073774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.073730    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj" event={"ID":"cf96195b-20f9-4bf4-80f6-6f9f06acc54d","Type":"ContainerDied","Data":"25d050e4f0b3bb9006e489b204829c69dadfaaedd01b73655630ddcca42beede"}
Sep 17 15:52:21.073774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.073747    2564 scope.go:117] "RemoveContainer" containerID="d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"
Sep 17 15:52:21.075045 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.075025    2564 generic.go:356] "Generic (PLEG): container finished" podID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" containerID="a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435" exitCode=0
Sep 17 15:52:21.075170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.075073    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"
Sep 17 15:52:21.075170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.075084    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" event={"ID":"d1e3eb54-331a-4b4b-a29e-1250b0ad4349","Type":"ContainerDied","Data":"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"}
Sep 17 15:52:21.075170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.075127    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4" event={"ID":"d1e3eb54-331a-4b4b-a29e-1250b0ad4349","Type":"ContainerDied","Data":"dd6150cb94e1ee7eefc345c58ddaa21f24e0076c2b7e37d25b36e79900bfee80"}
Sep 17 15:52:21.084528 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.084502    2564 scope.go:117] "RemoveContainer" containerID="d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"
Sep 17 15:52:21.084882 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:52:21.084854    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5\": container with ID starting with d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5 not found: ID does not exist" containerID="d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"
Sep 17 15:52:21.084965 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.084897    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5"} err="failed to get container status \"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5\": rpc error: code = NotFound desc = could not find container \"d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5\": container with ID starting with d112171af6a4010aa4b0851cae2e7e93612925672d209f19d3a312549a1fc8f5 not found: ID does not exist"
Sep 17 15:52:21.084965 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.084923    2564 scope.go:117] "RemoveContainer" containerID="a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"
Sep 17 15:52:21.093279 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.093261    2564 scope.go:117] "RemoveContainer" containerID="a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"
Sep 17 15:52:21.093527 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:52:21.093511    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435\": container with ID starting with a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435 not found: ID does not exist" containerID="a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"
Sep 17 15:52:21.093591 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.093535    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435"} err="failed to get container status \"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435\": rpc error: code = NotFound desc = could not find container \"a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435\": container with ID starting with a0f08d1698d494edc01e97ef81b267c092bdff4451b4d798d1c5a013abd0c435 not found: ID does not exist"
Sep 17 15:52:21.103404 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.103381    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:52:21.119540 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.119511    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-99719fde-5f7f756785-6xwlj"]
Sep 17 15:52:21.141565 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.141532    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:52:21.143700 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:21.143679    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-99719fde-7f4f8bcbf8-zdtq4"]
Sep 17 15:52:22.526082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:22.526054    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" path="/var/lib/kubelet/pods/cf96195b-20f9-4bf4-80f6-6f9f06acc54d/volumes"
Sep 17 15:52:22.526522 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:22.526392    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" path="/var/lib/kubelet/pods/d1e3eb54-331a-4b4b-a29e-1250b0ad4349/volumes"
Sep 17 15:52:22.526713 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:22.526701    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eabf247a-14ee-4356-b137-68b291137c51" path="/var/lib/kubelet/pods/eabf247a-14ee-4356-b137-68b291137c51/volumes"
Sep 17 15:52:29.550181 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:29.550136    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:52:44.569831 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:44.569783    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" podUID="55cc9579-f330-466e-9e3f-705156525b0d" containerName="main" containerID="cri-o://ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75" gracePeriod=45
Sep 17 15:52:44.823512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:44.823449    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:52:44.999889 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:44.999858    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:44.999949    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:44.999980    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000003    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000071 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000030    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000086    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vjvzc\" (UniqueName: \"kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc\") pod \"55cc9579-f330-466e-9e3f-705156525b0d\" (UID: \"55cc9579-f330-466e-9e3f-705156525b0d\") "
Sep 17 15:52:45.000293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000128    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:45.000293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000283    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home" (OuterVolumeSpecName: "home") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:45.000456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000331    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:45.000532 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000514    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.000655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000538    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.000655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.000553    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.002343 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.002314    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm" (OuterVolumeSpecName: "dshm") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:45.002478 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.002396    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:52:45.002603 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.002581    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc" (OuterVolumeSpecName: "kube-api-access-vjvzc") pod "55cc9579-f330-466e-9e3f-705156525b0d" (UID: "55cc9579-f330-466e-9e3f-705156525b0d"). InnerVolumeSpecName "kube-api-access-vjvzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:45.102024 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.101951    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55cc9579-f330-466e-9e3f-705156525b0d-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.102024 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.101976    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55cc9579-f330-466e-9e3f-705156525b0d-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.102024 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.101988    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjvzc\" (UniqueName: \"kubernetes.io/projected/55cc9579-f330-466e-9e3f-705156525b0d-kube-api-access-vjvzc\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:52:45.226197 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.226147    2564 generic.go:356] "Generic (PLEG): container finished" podID="55cc9579-f330-466e-9e3f-705156525b0d" containerID="ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75" exitCode=0
Sep 17 15:52:45.226347 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.226202    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"
Sep 17 15:52:45.226428 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.226411    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" event={"ID":"55cc9579-f330-466e-9e3f-705156525b0d","Type":"ContainerDied","Data":"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"}
Sep 17 15:52:45.226469 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.226450    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg" event={"ID":"55cc9579-f330-466e-9e3f-705156525b0d","Type":"ContainerDied","Data":"d03d2693f572e331daa7414163ff518923bcdd84107a96e7d62d8f1b9aef32ca"}
Sep 17 15:52:45.226509 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.226469    2564 scope.go:117] "RemoveContainer" containerID="ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"
Sep 17 15:52:45.236456 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.236437    2564 scope.go:117] "RemoveContainer" containerID="ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"
Sep 17 15:52:45.236772 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:52:45.236752    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75\": container with ID starting with ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75 not found: ID does not exist" containerID="ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"
Sep 17 15:52:45.236854 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.236787    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75"} err="failed to get container status \"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75\": rpc error: code = NotFound desc = could not find container \"ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75\": container with ID starting with ea46d5cd4c92784a0a5b2cb400f46bb5084dadab5ecc65eaa4c94933e5bf3b75 not found: ID does not exist"
Sep 17 15:52:45.255507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.255482    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:52:45.260118 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:45.260097    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-99719fde/e2e-lifecycle-model-99719fde-kserve-6d85c87f67-pdgrg"]
Sep 17 15:52:46.525854 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:46.525822    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cc9579-f330-466e-9e3f-705156525b0d" path="/var/lib/kubelet/pods/55cc9579-f330-466e-9e3f-705156525b0d/volumes"
Sep 17 15:52:47.796348 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:52:47.796317    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:53:00.365037 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365005    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365452    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cc9579-f330-466e-9e3f-705156525b0d" containerName="main"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365467    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cc9579-f330-466e-9e3f-705156525b0d" containerName="main"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365481    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" containerName="payload-pre-processing"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365488    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" containerName="payload-pre-processing"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365494    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eabf247a-14ee-4356-b137-68b291137c51" containerName="revoke-keys"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365500    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eabf247a-14ee-4356-b137-68b291137c51" containerName="revoke-keys"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365507    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" containerName="payload-processing"
Sep 17 15:53:00.365507 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365513    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" containerName="payload-processing"
Sep 17 15:53:00.365836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365582    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eabf247a-14ee-4356-b137-68b291137c51" containerName="revoke-keys"
Sep 17 15:53:00.365836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365593    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cc9579-f330-466e-9e3f-705156525b0d" containerName="main"
Sep 17 15:53:00.365836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365600    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf96195b-20f9-4bf4-80f6-6f9f06acc54d" containerName="payload-processing"
Sep 17 15:53:00.365836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.365608    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1e3eb54-331a-4b4b-a29e-1250b0ad4349" containerName="payload-pre-processing"
Sep 17 15:53:00.370147 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.370130    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.372879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.372857    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"kube-root-ca.crt\""
Sep 17 15:53:00.373594 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.373573    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"auth-test-model-5818c9-kserve-self-signed-certs\""
Sep 17 15:53:00.373724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.373707    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"default-dockercfg-dxwws\""
Sep 17 15:53:00.373774 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.373744    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"openshift-service-ca.crt\""
Sep 17 15:53:00.387052 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.387028    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:53:00.439517 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439493    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.439657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439530    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2jjn\" (UniqueName: \"kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.439657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439551    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.439657 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439629    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.439775 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439678    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.439775 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.439710    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540626 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540586    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540641    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2jjn\" (UniqueName: \"kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540664    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540680    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540716    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.540803 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540751    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.541001 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.540981    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.541244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.541225    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.541323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.541248    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.542954 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.542937    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.543385 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.543363    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.550931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.550906    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2jjn\" (UniqueName: \"kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn\") pod \"auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.679751 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.679662    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:00.957048 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:00.957018    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:53:00.957989 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:53:00.957963    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4107d15b_fd3c_4915_a9b0_d971f8d0ae7f.slice/crio-5c1069d94131742d84d82c92a830c33469d12e843ba2f9a6a08d7761b7a3b1a3 WatchSource:0}: Error finding container 5c1069d94131742d84d82c92a830c33469d12e843ba2f9a6a08d7761b7a3b1a3: Status 404 returned error can't find the container with id 5c1069d94131742d84d82c92a830c33469d12e843ba2f9a6a08d7761b7a3b1a3
Sep 17 15:53:01.322148 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:01.322108    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" event={"ID":"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f","Type":"ContainerStarted","Data":"640909a5bd1901bc1794bd48913242d77e915bafa298207d77ce1846e6b77752"}
Sep 17 15:53:01.322322 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:01.322156    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" event={"ID":"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f","Type":"ContainerStarted","Data":"5c1069d94131742d84d82c92a830c33469d12e843ba2f9a6a08d7761b7a3b1a3"}
Sep 17 15:53:01.343648 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:01.343573    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" podStartSLOduration=1.3435564150000001 podStartE2EDuration="1.343556415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:01.340342157 +0000 UTC m=+1373.307519975" watchObservedRunningTime="2026-09-17 15:53:01.343556415 +0000 UTC m=+1373.310734233"
Sep 17 15:53:09.369269 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.369231    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:53:09.373829 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.373808    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.376396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.376376    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-849b71\"/\"embedding-w0-849b71-kserve-self-signed-certs\""
Sep 17 15:53:09.387105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.387083    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:53:09.432331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432308    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7r4c\" (UniqueName: \"kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.432448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432350    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.432448 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432387    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.432528 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432450    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.432528 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432472    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.432528 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.432497    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.533859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.533833    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.533859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.533859    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.533886    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.533943    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7r4c\" (UniqueName: \"kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.533971    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.534019    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534282 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.534260    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.534273    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.534400 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.534379    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.536197 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.536175    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.536605 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.536585    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.542819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.542801    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7r4c\" (UniqueName: \"kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c\") pod \"embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") " pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.683168 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.683088    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:09.994051 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:09.994028    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:53:09.996364 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:53:09.996327    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ba5e0bf_db72_4895_93f6_71701d082eb9.slice/crio-b37dc96e3af324b3407730e2da570c8ff27d9f815bdad68c08c55ba2d6d15fe3 WatchSource:0}: Error finding container b37dc96e3af324b3407730e2da570c8ff27d9f815bdad68c08c55ba2d6d15fe3: Status 404 returned error can't find the container with id b37dc96e3af324b3407730e2da570c8ff27d9f815bdad68c08c55ba2d6d15fe3
Sep 17 15:53:10.383245 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.383208    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" event={"ID":"1ba5e0bf-db72-4895-93f6-71701d082eb9","Type":"ContainerStarted","Data":"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"}
Sep 17 15:53:10.383684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.383257    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" event={"ID":"1ba5e0bf-db72-4895-93f6-71701d082eb9","Type":"ContainerStarted","Data":"b37dc96e3af324b3407730e2da570c8ff27d9f815bdad68c08c55ba2d6d15fe3"}
Sep 17 15:53:10.414771 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.414711    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" podStartSLOduration=1.414690008 podStartE2EDuration="1.414690008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:10.413724973 +0000 UTC m=+1382.380902791" watchObservedRunningTime="2026-09-17 15:53:10.414690008 +0000 UTC m=+1382.381867825"
Sep 17 15:53:10.680504 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.680408    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:10.680504 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.680474    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:10.693235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:10.693209    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:11.406648 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:11.406574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:53:19.684082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:19.684044    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:19.684499 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:19.684135    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:19.696281 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:19.696254    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:20.456474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:20.456440    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:53:33.303175 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:53:33.303132    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:54:08.876086 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:08.876001    2564 scope.go:117] "RemoveContainer" containerID="328eac68ca7810d0d15e2ac7a66a44e1c89a2ad19cb5febf13786ad1fa2c6f10"
Sep 17 15:54:11.786374 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:11.786337    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:54:19.904249 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:19.904215    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"]
Sep 17 15:54:19.909431 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:19.909411    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:19.912484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:19.912452    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:54:19.912668 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:19.912648    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:54:19.920601 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:19.920575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"]
Sep 17 15:54:20.043029 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.042988    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k\" (UID: \"a2b802d9-0379-477f-80f6-73812611dffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:20.144582 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.144549    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k\" (UID: \"a2b802d9-0379-477f-80f6-73812611dffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:20.145298 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.145274    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k\" (UID: \"a2b802d9-0379-477f-80f6-73812611dffa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:20.221022 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.220935    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:20.496798 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.496773    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"]
Sep 17 15:54:20.498437 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:54:20.498408    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2b802d9_0379_477f_80f6_73812611dffa.slice/crio-d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b WatchSource:0}: Error finding container d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b: Status 404 returned error can't find the container with id d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b
Sep 17 15:54:20.799940 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:20.799896    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k" event={"ID":"a2b802d9-0379-477f-80f6-73812611dffa","Type":"ContainerStarted","Data":"d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b"}
Sep 17 15:54:21.809390 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:21.809353    2564 generic.go:356] "Generic (PLEG): container finished" podID="a2b802d9-0379-477f-80f6-73812611dffa" containerID="9790abc0efbcdbfe7b8e8f3721821447c4cab29eccd7443737162f8283bbca99" exitCode=0
Sep 17 15:54:21.809854 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:21.809449    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k" event={"ID":"a2b802d9-0379-477f-80f6-73812611dffa","Type":"ContainerDied","Data":"9790abc0efbcdbfe7b8e8f3721821447c4cab29eccd7443737162f8283bbca99"}
Sep 17 15:54:22.964061 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:22.964033    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:23.076592 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.076520    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca\") pod \"a2b802d9-0379-477f-80f6-73812611dffa\" (UID: \"a2b802d9-0379-477f-80f6-73812611dffa\") "
Sep 17 15:54:23.076885 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.076861    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a2b802d9-0379-477f-80f6-73812611dffa" (UID: "a2b802d9-0379-477f-80f6-73812611dffa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:23.178163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.178133    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2b802d9-0379-477f-80f6-73812611dffa-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:23.822852 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.822815    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"
Sep 17 15:54:23.823035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.822856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k" event={"ID":"a2b802d9-0379-477f-80f6-73812611dffa","Type":"ContainerDied","Data":"d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b"}
Sep 17 15:54:23.823035 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:23.822914    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5cb73ccc361ee2cc392038ecaeb1ff5a0f203269182d0976b1fc529aba1618b"
Sep 17 15:54:24.831427 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:24.831387    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:54:34.263945 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.263909    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"]
Sep 17 15:54:34.264396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.264374    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2b802d9-0379-477f-80f6-73812611dffa" containerName="revoke-keys"
Sep 17 15:54:34.264396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.264391    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2b802d9-0379-477f-80f6-73812611dffa" containerName="revoke-keys"
Sep 17 15:54:34.264547 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.264453    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2b802d9-0379-477f-80f6-73812611dffa" containerName="revoke-keys"
Sep 17 15:54:34.267630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.267600    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:34.269932 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.269906    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 15:54:34.270044 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.269944    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 15:54:34.276250 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.276224    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"]
Sep 17 15:54:34.392968 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.392934    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75\" (UID: \"179b7682-6647-4add-831d-76aebaff1aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:34.494059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.494023    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75\" (UID: \"179b7682-6647-4add-831d-76aebaff1aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:34.494729 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.494705    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75\" (UID: \"179b7682-6647-4add-831d-76aebaff1aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:34.579475 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.579394    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:34.863028 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.862818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"]
Sep 17 15:54:34.866072 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:54:34.866041    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod179b7682_6647_4add_831d_76aebaff1aee.slice/crio-866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42 WatchSource:0}: Error finding container 866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42: Status 404 returned error can't find the container with id 866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42
Sep 17 15:54:34.898422 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:34.898392    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75" event={"ID":"179b7682-6647-4add-831d-76aebaff1aee","Type":"ContainerStarted","Data":"866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42"}
Sep 17 15:54:35.906216 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:35.906183    2564 generic.go:356] "Generic (PLEG): container finished" podID="179b7682-6647-4add-831d-76aebaff1aee" containerID="c66bed2ccc86c68cf2c55e6db5a47b6863093bbe806a67beb90e3f51d56eca79" exitCode=0
Sep 17 15:54:35.906597 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:35.906270    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75" event={"ID":"179b7682-6647-4add-831d-76aebaff1aee","Type":"ContainerDied","Data":"c66bed2ccc86c68cf2c55e6db5a47b6863093bbe806a67beb90e3f51d56eca79"}
Sep 17 15:54:37.054935 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.054911    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:37.223455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.223376    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca\") pod \"179b7682-6647-4add-831d-76aebaff1aee\" (UID: \"179b7682-6647-4add-831d-76aebaff1aee\") "
Sep 17 15:54:37.223827 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.223788    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "179b7682-6647-4add-831d-76aebaff1aee" (UID: "179b7682-6647-4add-831d-76aebaff1aee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:37.324288 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.324258    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/179b7682-6647-4add-831d-76aebaff1aee-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:37.921996 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.921966    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"
Sep 17 15:54:37.922184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.921957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75" event={"ID":"179b7682-6647-4add-831d-76aebaff1aee","Type":"ContainerDied","Data":"866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42"}
Sep 17 15:54:37.922184 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:37.922106    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866842e6c659a3ccb65f6c11a0db6cc7a5f81cea2cd28813ffa6b2b2a33e9f42"
Sep 17 15:54:38.494287 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:38.494244    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"]
Sep 17 15:54:38.496529 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:38.496504    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c86e9a-29827665-76kzl"]
Sep 17 15:54:38.531579 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:38.531544    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161e5807-e23c-4597-80d5-67a1363f1186" path="/var/lib/kubelet/pods/161e5807-e23c-4597-80d5-67a1363f1186/volumes"
Sep 17 15:54:38.907242 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:38.907204    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"]
Sep 17 15:54:38.912562 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:38.912527    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c86e9a-r6v9k"]
Sep 17 15:54:39.130452 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:39.130415    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:54:39.164105 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:39.164026    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:54:39.850828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:39.850786    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" podUID="22c95a57-a6c1-46d3-baab-87c4b01ff856" containerName="main" containerID="cri-o://6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd" gracePeriod=45
Sep 17 15:54:40.102556 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.102497    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:54:40.253772 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253737    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.253942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253790    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hbw9\" (UniqueName: \"kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.253942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.253942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253854    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.253942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253876    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.253942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.253923    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache\") pod \"22c95a57-a6c1-46d3-baab-87c4b01ff856\" (UID: \"22c95a57-a6c1-46d3-baab-87c4b01ff856\") "
Sep 17 15:54:40.254219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254013    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.254219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254189    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home" (OuterVolumeSpecName: "home") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.254344 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254225    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache" (OuterVolumeSpecName: "model-cache") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.254432 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254412    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.254491 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254434    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.254491 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.254474    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.256069 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.256046    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm" (OuterVolumeSpecName: "dshm") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:40.256183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.256088    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9" (OuterVolumeSpecName: "kube-api-access-8hbw9") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "kube-api-access-8hbw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:40.256183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.256126    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22c95a57-a6c1-46d3-baab-87c4b01ff856" (UID: "22c95a57-a6c1-46d3-baab-87c4b01ff856"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:40.355139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.355047    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbw9\" (UniqueName: \"kubernetes.io/projected/22c95a57-a6c1-46d3-baab-87c4b01ff856-kube-api-access-8hbw9\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.355139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.355079    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22c95a57-a6c1-46d3-baab-87c4b01ff856-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.355139 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.355089    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22c95a57-a6c1-46d3-baab-87c4b01ff856-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:40.526233 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.526201    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2b802d9-0379-477f-80f6-73812611dffa" path="/var/lib/kubelet/pods/a2b802d9-0379-477f-80f6-73812611dffa/volumes"
Sep 17 15:54:40.939878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.939843    2564 generic.go:356] "Generic (PLEG): container finished" podID="22c95a57-a6c1-46d3-baab-87c4b01ff856" containerID="6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd" exitCode=0
Sep 17 15:54:40.940301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.939889    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"
Sep 17 15:54:40.940301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.939926    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" event={"ID":"22c95a57-a6c1-46d3-baab-87c4b01ff856","Type":"ContainerDied","Data":"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"}
Sep 17 15:54:40.940301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.939968    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm" event={"ID":"22c95a57-a6c1-46d3-baab-87c4b01ff856","Type":"ContainerDied","Data":"7516ca5e9b00998d2b7220f8e556b476bfc6e2b3792a12b0e7ee5e68ebea14c1"}
Sep 17 15:54:40.940301 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.939986    2564 scope.go:117] "RemoveContainer" containerID="6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"
Sep 17 15:54:40.949306 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.949291    2564 scope.go:117] "RemoveContainer" containerID="6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"
Sep 17 15:54:40.949566 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:54:40.949548    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd\": container with ID starting with 6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd not found: ID does not exist" containerID="6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"
Sep 17 15:54:40.949651 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.949575    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd"} err="failed to get container status \"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd\": rpc error: code = NotFound desc = could not find container \"6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd\": container with ID starting with 6df1eb8e06ad6b6f9251f3d82506181334131c0ca6e59563a717d010012c6fdd not found: ID does not exist"
Sep 17 15:54:40.960996 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.960968    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:54:40.967567 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:40.967546    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c86e9a/distinct-w4-c86e9a-kserve-7f967b7bc-2k5hm"]
Sep 17 15:54:42.085166 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:42.085136    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:54:42.525757 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:42.525727    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22c95a57-a6c1-46d3-baab-87c4b01ff856" path="/var/lib/kubelet/pods/22c95a57-a6c1-46d3-baab-87c4b01ff856/volumes"
Sep 17 15:54:46.145377 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:46.145340    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:54:46.145754 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:46.145593    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" containerID="cri-o://8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1" gracePeriod=30
Sep 17 15:54:50.078176 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:50.078137    2564 patch_prober.go:28] interesting pod/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 15:54:50.078543 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:50.078197    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:51.403553 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.403530    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:54:51.457833 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457749    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457835    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457856    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457874    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4htww\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.457994 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.457951    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.458323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458015    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.458323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458053    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket\") pod \"11685c29-4640-442d-a39c-94291e185ea7\" (UID: \"11685c29-4640-442d-a39c-94291e185ea7\") "
Sep 17 15:54:51.458323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.458323 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458277    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:51.458546 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458486    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.458546 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458540    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11685c29-4640-442d-a39c-94291e185ea7-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.458685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458568    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.458685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458536    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data" (OuterVolumeSpecName: "istio-data") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.458685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.458642    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.460088 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.460051    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:54:51.460190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.460136    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:51.460190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.460157    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww" (OuterVolumeSpecName: "kube-api-access-4htww") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "kube-api-access-4htww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:51.460190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.460175    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token" (OuterVolumeSpecName: "istio-token") pod "11685c29-4640-442d-a39c-94291e185ea7" (UID: "11685c29-4640-442d-a39c-94291e185ea7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:51.559059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559028    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11685c29-4640-442d-a39c-94291e185ea7-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559059 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559055    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559067    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559075    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559084    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559092    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11685c29-4640-442d-a39c-94291e185ea7-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:51.559283 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:51.559100    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htww\" (UniqueName: \"kubernetes.io/projected/11685c29-4640-442d-a39c-94291e185ea7-kube-api-access-4htww\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:52.011673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.011631    2564 generic.go:356] "Generic (PLEG): container finished" podID="11685c29-4640-442d-a39c-94291e185ea7" containerID="8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1" exitCode=0
Sep 17 15:54:52.011673 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.011672    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"
Sep 17 15:54:52.011898 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.011714    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerDied","Data":"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"}
Sep 17 15:54:52.011898 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.011781    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298" event={"ID":"11685c29-4640-442d-a39c-94291e185ea7","Type":"ContainerDied","Data":"1a4e684b81e37d11ab398082fcee1906edc3c9c178deabe52b797213c1fb446a"}
Sep 17 15:54:52.011898 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.011812    2564 scope.go:117] "RemoveContainer" containerID="8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"
Sep 17 15:54:52.021555 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.021539    2564 scope.go:117] "RemoveContainer" containerID="6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"
Sep 17 15:54:52.029766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.029749    2564 scope.go:117] "RemoveContainer" containerID="8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"
Sep 17 15:54:52.030047 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:54:52.030010    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1\": container with ID starting with 8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1 not found: ID does not exist" containerID="8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"
Sep 17 15:54:52.030085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.030055    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1"} err="failed to get container status \"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1\": rpc error: code = NotFound desc = could not find container \"8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1\": container with ID starting with 8b64fdaa0ff2a376d0ad5a17e35193f1bf5a68d14f1a210ccdc7932920c245a1 not found: ID does not exist"
Sep 17 15:54:52.030085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.030073    2564 scope.go:117] "RemoveContainer" containerID="6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"
Sep 17 15:54:52.030323 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:54:52.030304    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108\": container with ID starting with 6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108 not found: ID does not exist" containerID="6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"
Sep 17 15:54:52.030369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.030330    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108"} err="failed to get container status \"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108\": rpc error: code = NotFound desc = could not find container \"6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108\": container with ID starting with 6750add593be5d127ddd1f12e37517ca80d0f8ff68ec1959c49009f87d8a2108 not found: ID does not exist"
Sep 17 15:54:52.034825 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.034804    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:54:52.038808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.038787    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c86e9a-openshift-default-f7dc6cb5b-p4298"]
Sep 17 15:54:52.525864 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.525827    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11685c29-4640-442d-a39c-94291e185ea7" path="/var/lib/kubelet/pods/11685c29-4640-442d-a39c-94291e185ea7/volumes"
Sep 17 15:54:52.681596 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.681553    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"]
Sep 17 15:54:52.685838 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.685811    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-849b71-29827665-d47gs"]
Sep 17 15:54:52.901260 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.901175    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"]
Sep 17 15:54:52.904140 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:52.904112    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-849b71-cfl75"]
Sep 17 15:54:54.149725 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.149686    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerName="main" containerID="cri-o://c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217" gracePeriod=45
Sep 17 15:54:54.182487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.182452    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" podUID="1ba5e0bf-db72-4895-93f6-71701d082eb9" containerName="main" containerID="cri-o://a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb" gracePeriod=45
Sep 17 15:54:54.256011 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.255979    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/ready\": dial tcp 10.133.0.78:8000: connect: connection refused"
Sep 17 15:54:54.464579 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.464557    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:54:54.467766 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.467746    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:54:54.527900 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.527866    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="179b7682-6647-4add-831d-76aebaff1aee" path="/var/lib/kubelet/pods/179b7682-6647-4add-831d-76aebaff1aee/volumes"
Sep 17 15:54:54.528455 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.528281    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed50bed4-416a-4012-86a5-8f072c0c4b4d" path="/var/lib/kubelet/pods/ed50bed4-416a-4012-86a5-8f072c0c4b4d/volumes"
Sep 17 15:54:54.587655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587589    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.587819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587666    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7r4c\" (UniqueName: \"kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.587819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587730    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.587819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587773    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.587819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587799    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h48bl\" (UniqueName: \"kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.587819 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587814    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587837    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587859    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587879    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587907    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587921    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm\") pod \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\" (UID: \"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.587936    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home\") pod \"1ba5e0bf-db72-4895-93f6-71701d082eb9\" (UID: \"1ba5e0bf-db72-4895-93f6-71701d082eb9\") "
Sep 17 15:54:54.588096 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.588018    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.588484 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.588365    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.589296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.588558    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home" (OuterVolumeSpecName: "home") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.589296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.588770    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.589296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.588968    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home" (OuterVolumeSpecName: "home") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.589296 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.589222    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.589603 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.589302    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.590502 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.590470    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:54.591452 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.591404    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm" (OuterVolumeSpecName: "dshm") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.591542 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.591475    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl" (OuterVolumeSpecName: "kube-api-access-h48bl") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "kube-api-access-h48bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:54.591693 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.591666    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c" (OuterVolumeSpecName: "kube-api-access-x7r4c") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "kube-api-access-x7r4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:54.591772 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.591756    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" (UID: "d88a93c1-9fe5-4579-a09a-60b2ad14fbe3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:54.591815 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.591775    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm" (OuterVolumeSpecName: "dshm") pod "1ba5e0bf-db72-4895-93f6-71701d082eb9" (UID: "1ba5e0bf-db72-4895-93f6-71701d082eb9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:54.689203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689168    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689196    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ba5e0bf-db72-4895-93f6-71701d082eb9-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689206    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7r4c\" (UniqueName: \"kubernetes.io/projected/1ba5e0bf-db72-4895-93f6-71701d082eb9-kube-api-access-x7r4c\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689216    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689226    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h48bl\" (UniqueName: \"kubernetes.io/projected/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-kube-api-access-h48bl\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689234    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689242    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ba5e0bf-db72-4895-93f6-71701d082eb9-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689250    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689257    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689264    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:54.689471 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:54.689273    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:54:55.034490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.034451    2564 generic.go:356] "Generic (PLEG): container finished" podID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerID="c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217" exitCode=0
Sep 17 15:54:55.034684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.034508    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"
Sep 17 15:54:55.034684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.034541    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" event={"ID":"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3","Type":"ContainerDied","Data":"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"}
Sep 17 15:54:55.034684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.034583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g" event={"ID":"d88a93c1-9fe5-4579-a09a-60b2ad14fbe3","Type":"ContainerDied","Data":"c1346165375b77242cdaa8917720055bc5d80fbb5bb08a3f9c85e688a422ea90"}
Sep 17 15:54:55.034684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.034601    2564 scope.go:117] "RemoveContainer" containerID="c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"
Sep 17 15:54:55.035996 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.035971    2564 generic.go:356] "Generic (PLEG): container finished" podID="1ba5e0bf-db72-4895-93f6-71701d082eb9" containerID="a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb" exitCode=0
Sep 17 15:54:55.036097 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.036021    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"
Sep 17 15:54:55.036097 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.036064    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" event={"ID":"1ba5e0bf-db72-4895-93f6-71701d082eb9","Type":"ContainerDied","Data":"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"}
Sep 17 15:54:55.036203 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.036104    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b" event={"ID":"1ba5e0bf-db72-4895-93f6-71701d082eb9","Type":"ContainerDied","Data":"b37dc96e3af324b3407730e2da570c8ff27d9f815bdad68c08c55ba2d6d15fe3"}
Sep 17 15:54:55.045300 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.045284    2564 scope.go:117] "RemoveContainer" containerID="c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"
Sep 17 15:54:55.045587 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:54:55.045568    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217\": container with ID starting with c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217 not found: ID does not exist" containerID="c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"
Sep 17 15:54:55.045683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.045599    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217"} err="failed to get container status \"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217\": rpc error: code = NotFound desc = could not find container \"c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217\": container with ID starting with c59be8184a8673d6c6c25dbfc6fa6833a9153365f417cdfb7732b599c7ea6217 not found: ID does not exist"
Sep 17 15:54:55.045683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.045638    2564 scope.go:117] "RemoveContainer" containerID="a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"
Sep 17 15:54:55.054270 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.054251    2564 scope.go:117] "RemoveContainer" containerID="a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"
Sep 17 15:54:55.054532 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:54:55.054515    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb\": container with ID starting with a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb not found: ID does not exist" containerID="a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"
Sep 17 15:54:55.054586 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.054538    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb"} err="failed to get container status \"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb\": rpc error: code = NotFound desc = could not find container \"a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb\": container with ID starting with a1fd43196750e04b5fe2d92f19012477a21c7c9e256333b69c33b175f9ed6ddb not found: ID does not exist"
Sep 17 15:54:55.067219 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.067187    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:54:55.072235 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.072215    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/embedding-w0-849b71-kserve-7b7c6f44f7-xqq8b"]
Sep 17 15:54:55.098825 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.098774    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:54:55.102308 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:55.102289    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-849b71/distinct-2-w0-849b71-kserve-7c95cccbc-x4m8g"]
Sep 17 15:54:56.526286 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:56.526251    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ba5e0bf-db72-4895-93f6-71701d082eb9" path="/var/lib/kubelet/pods/1ba5e0bf-db72-4895-93f6-71701d082eb9/volumes"
Sep 17 15:54:56.526704 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:56.526689    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" path="/var/lib/kubelet/pods/d88a93c1-9fe5-4579-a09a-60b2ad14fbe3/volumes"
Sep 17 15:54:57.426971 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.426937    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:54:57.427370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427358    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerName="main"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427371    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerName="main"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427383    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427390    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427398    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427404    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427411    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ba5e0bf-db72-4895-93f6-71701d082eb9" containerName="main"
Sep 17 15:54:57.427419 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427416    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ba5e0bf-db72-4895-93f6-71701d082eb9" containerName="main"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427425    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="179b7682-6647-4add-831d-76aebaff1aee" containerName="revoke-keys"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427430    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="179b7682-6647-4add-831d-76aebaff1aee" containerName="revoke-keys"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427437    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22c95a57-a6c1-46d3-baab-87c4b01ff856" containerName="main"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427442    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="22c95a57-a6c1-46d3-baab-87c4b01ff856" containerName="main"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427500    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427507    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="179b7682-6647-4add-831d-76aebaff1aee" containerName="revoke-keys"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427515    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d88a93c1-9fe5-4579-a09a-60b2ad14fbe3" containerName="main"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427522    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ba5e0bf-db72-4895-93f6-71701d082eb9" containerName="main"
Sep 17 15:54:57.427666 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.427527    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="22c95a57-a6c1-46d3-baab-87c4b01ff856" containerName="main"
Sep 17 15:54:57.432058 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.432041    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.440693 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.440672    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8f99a1-openshift-default-dockercfg-cgm72\""
Sep 17 15:54:57.450474 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.450451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526351    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526415    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526515    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526553    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcvt\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526681    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526732    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526761    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.526846 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.526791    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628121    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcvt\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628173    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628223    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628258    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628328 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628300    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628508 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628332    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628508 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628378    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628508 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628465    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628508 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628504    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628759 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628713    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628810 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628757    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628874    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.628960 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.628936    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.629157 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.629141    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.630644 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.630602    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.630837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.630822    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.639009 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.638971    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcvt\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.639009 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.638995    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token\") pod \"e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") " pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:57.743022 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:57.742933    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:58.027695 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:58.027667    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:54:58.029917 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:54:58.029891    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccedd1d_af00_43f4_8c59_b7f679c36092.slice/crio-b7490073b4192d917c0e14a3423bab3e51a21b06465c799812b78cc3bc57976e WatchSource:0}: Error finding container b7490073b4192d917c0e14a3423bab3e51a21b06465c799812b78cc3bc57976e: Status 404 returned error can't find the container with id b7490073b4192d917c0e14a3423bab3e51a21b06465c799812b78cc3bc57976e
Sep 17 15:54:58.032688 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:58.032650    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:54:58.032763 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:58.032737    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:54:58.032801 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:58.032789    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:54:58.062578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:58.062550    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerStarted","Data":"b7490073b4192d917c0e14a3423bab3e51a21b06465c799812b78cc3bc57976e"}
Sep 17 15:54:59.069337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:59.069297    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerStarted","Data":"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"}
Sep 17 15:54:59.743369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:59.743338    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:59.748599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:59.748574    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:54:59.777965 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:54:59.777922    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podStartSLOduration=2.777909259 podStartE2EDuration="2.777909259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:59.105682328 +0000 UTC m=+1491.072860145" watchObservedRunningTime="2026-09-17 15:54:59.777909259 +0000 UTC m=+1491.745087075"
Sep 17 15:55:00.075681 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:00.075651    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:00.076486 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:00.076467    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:08.896808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:08.896774    2564 scope.go:117] "RemoveContainer" containerID="7497f630a349e4ada719245abda0089fe9b15f185a37cc873f6e0a338baa980b"
Sep 17 15:55:08.905131 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:08.905110    2564 scope.go:117] "RemoveContainer" containerID="e4ae531c92bf07b133b536e40bb70799893de48c4f08f2aad8029135ce83a4f9"
Sep 17 15:55:08.914345 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:08.914322    2564 scope.go:117] "RemoveContainer" containerID="b05ffae186f709ff5c5c3085379f124d3ae6d99a8935472f315ff63b4827b210"
Sep 17 15:55:08.921816 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:08.921796    2564 scope.go:117] "RemoveContainer" containerID="901b87542f8f9ce6e91eaee4888a8561e50086171576b6d59addef39f496175d"
Sep 17 15:55:22.111760 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:22.111721    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:55:31.117878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:31.117789    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 15:55:31.118250 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:31.117874    2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 15:55:33.275003 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.274969    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerID="c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266" exitCode=0
Sep 17 15:55:33.275435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.275043    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerDied","Data":"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"}
Sep 17 15:55:33.275435 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.275415    2564 scope.go:117] "RemoveContainer" containerID="c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"
Sep 17 15:55:33.276527 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.276501    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:55:33.276578 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.276567    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:55:33.276641 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:33.276599    2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236204Ki","pods":"250"}
Sep 17 15:55:34.288252 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:34.288216    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerStarted","Data":"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"}
Sep 17 15:55:34.743511 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:34.743417    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:34.743772 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:34.743752    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 17 15:55:34.743831 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:34.743800    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 17 15:55:35.743446 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:35.743409    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 17 15:55:35.743893 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:35.743468    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 17 15:55:36.744170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:36.744137    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 17 15:55:36.744655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:36.744204    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 17 15:55:37.744190 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:37.744147    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 17 15:55:37.744686 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:37.744210    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 17 15:55:38.744246 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:38.744207    2564 patch_prober.go:28] interesting pod/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Sep 17 15:55:38.744674 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:38.744269    2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Sep 17 15:55:39.747185 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:39.747150    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:39.747694 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:39.747431    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:39.748159 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:39.748141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:55:43.790151 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:55:43.790116    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:56:02.716751 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:02.716713    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:56:08.975183 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:08.975153    2564 scope.go:117] "RemoveContainer" containerID="7f6498e0987868e9a807a1cea38f3b6dc5c0425b7c9dfbb5b4f8ba54a281286f"
Sep 17 15:56:19.954795 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:19.954758    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:56:24.892512 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:24.892473    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:56:24.893030 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:24.892761    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy" containerID="cri-o://7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0" gracePeriod=30
Sep 17 15:56:30.143953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.143926    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:56:30.196930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.196861    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.196930 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.196895    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.196948    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.196988    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197044    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197129 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197094    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197107    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197200    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197242    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197255    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197262    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qcvt\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197290    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:30.197356 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197328    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy\") pod \"3ccedd1d-af00-43f4-8c59-b7f679c36092\" (UID: \"3ccedd1d-af00-43f4-8c59-b7f679c36092\") "
Sep 17 15:56:30.197770 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197472    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:30.197770 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197763    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.197888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197784    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.197888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197798    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.197888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197811    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ccedd1d-af00-43f4-8c59-b7f679c36092-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.197888 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.197824    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.199378 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.199352    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:30.199787 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.199761    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:56:30.199878 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.199795    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt" (OuterVolumeSpecName: "kube-api-access-2qcvt") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "kube-api-access-2qcvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:30.199921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.199886    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ccedd1d-af00-43f4-8c59-b7f679c36092" (UID: "3ccedd1d-af00-43f4-8c59-b7f679c36092"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:30.298959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.298921    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.298959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.298953    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcvt\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-kube-api-access-2qcvt\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.298959 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.298967    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.299196 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.298979    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ccedd1d-af00-43f4-8c59-b7f679c36092-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:30.619244 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.619213    2564 generic.go:356] "Generic (PLEG): container finished" podID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerID="7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0" exitCode=0
Sep 17 15:56:30.619401 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.619259    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"
Sep 17 15:56:30.619401 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.619291    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerDied","Data":"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"}
Sep 17 15:56:30.619401 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.619342    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd" event={"ID":"3ccedd1d-af00-43f4-8c59-b7f679c36092","Type":"ContainerDied","Data":"b7490073b4192d917c0e14a3423bab3e51a21b06465c799812b78cc3bc57976e"}
Sep 17 15:56:30.619401 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.619367    2564 scope.go:117] "RemoveContainer" containerID="7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"
Sep 17 15:56:30.628826 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.628810    2564 scope.go:117] "RemoveContainer" containerID="c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"
Sep 17 15:56:30.636824 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.636810    2564 scope.go:117] "RemoveContainer" containerID="7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"
Sep 17 15:56:30.637096 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:30.637077    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0\": container with ID starting with 7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0 not found: ID does not exist" containerID="7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"
Sep 17 15:56:30.637144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.637103    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0"} err="failed to get container status \"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0\": rpc error: code = NotFound desc = could not find container \"7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0\": container with ID starting with 7674c7a124e762f126b13dadb75580aa5e91aff0554e3cc4dd24d3ad55ac33a0 not found: ID does not exist"
Sep 17 15:56:30.637144 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.637120    2564 scope.go:117] "RemoveContainer" containerID="c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"
Sep 17 15:56:30.637336 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:30.637318    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266\": container with ID starting with c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266 not found: ID does not exist" containerID="c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"
Sep 17 15:56:30.637396 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.637345    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266"} err="failed to get container status \"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266\": rpc error: code = NotFound desc = could not find container \"c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266\": container with ID starting with c6b1d0b034627aabbaa474bfde8dfbd2a303aeb9ff23e3e9c90f4b96a1e6f266 not found: ID does not exist"
Sep 17 15:56:30.641501 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.641466    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:56:30.643706 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:30.643683    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8f99a1-openshift-default-6bcdbf8f66-4qmrd"]
Sep 17 15:56:32.525195 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:32.525161    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" path="/var/lib/kubelet/pods/3ccedd1d-af00-43f4-8c59-b7f679c36092/volumes"
Sep 17 15:56:35.399655 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:35.399595    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:56:36.216054 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.216019    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:56:36.216337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.216271    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" podUID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" containerName="payload-processing" containerID="cri-o://458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929" gracePeriod=30
Sep 17 15:56:36.468961 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.468905    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:56:36.557983 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.557945    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume\") pod \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") "
Sep 17 15:56:36.558176 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.557991    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk6dq\" (UniqueName: \"kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq\") pod \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\" (UID: \"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba\") "
Sep 17 15:56:36.558754 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.558725    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" (UID: "a726c89e-61ff-4e83-8ecc-6fb8b13bdfba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:36.560238 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.560198    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq" (OuterVolumeSpecName: "kube-api-access-gk6dq") pod "a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" (UID: "a726c89e-61ff-4e83-8ecc-6fb8b13bdfba"). InnerVolumeSpecName "kube-api-access-gk6dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:36.659164 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.659129    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:36.659164 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.659164    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk6dq\" (UniqueName: \"kubernetes.io/projected/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba-kube-api-access-gk6dq\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:36.660503 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.660470    2564 generic.go:356] "Generic (PLEG): container finished" podID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" containerID="458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929" exitCode=0
Sep 17 15:56:36.660678 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.660517    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" event={"ID":"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba","Type":"ContainerDied","Data":"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"}
Sep 17 15:56:36.660764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.660533    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"
Sep 17 15:56:36.660764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.660732    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw" event={"ID":"a726c89e-61ff-4e83-8ecc-6fb8b13bdfba","Type":"ContainerDied","Data":"9b6574cad94ba0d4ea484e384aa23127b01099be45ce71ea89ac1117ccb0c55c"}
Sep 17 15:56:36.660872 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.660771    2564 scope.go:117] "RemoveContainer" containerID="458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"
Sep 17 15:56:36.674583 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.674560    2564 scope.go:117] "RemoveContainer" containerID="458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"
Sep 17 15:56:36.674956 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:36.674936    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929\": container with ID starting with 458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929 not found: ID does not exist" containerID="458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"
Sep 17 15:56:36.675024 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.674965    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929"} err="failed to get container status \"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929\": rpc error: code = NotFound desc = could not find container \"458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929\": container with ID starting with 458e3f5d7750d140ad2557ea8d628785f13aa2ecf99f1a9867e9c5814477b929 not found: ID does not exist"
Sep 17 15:56:36.703709 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.703680    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:56:36.707556 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:36.707521    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-2b23c7-5c986fff67-kxdmw"]
Sep 17 15:56:38.525684 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:38.525651    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" path="/var/lib/kubelet/pods/a726c89e-61ff-4e83-8ecc-6fb8b13bdfba/volumes"
Sep 17 15:56:42.602953 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:42.602918    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:56:44.658936 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:44.658900    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:56:44.659392 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:44.659152    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy" containerID="cri-o://0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c" gracePeriod=30
Sep 17 15:56:48.530701 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:48.530670    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:56:49.916731 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:49.916707    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:56:50.098909 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.098873    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.098939    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.098993    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099027    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099101 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099055    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099120    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099151    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8dsm\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099227    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099314 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099250    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data\") pod \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\" (UID: \"9243c8a3-eb3c-49c4-bdc6-906984e7af60\") "
Sep 17 15:56:50.099514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099301    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:50.099514 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099387    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:50.099697 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099673    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:50.099810 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099768    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:50.099957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099890    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.099957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099918    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.099957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099932    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.099957 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.099944    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.100269 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.100249    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data" (OuterVolumeSpecName: "istio-data") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:50.101291 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.101269    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 15:56:50.101370 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.101285    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token" (OuterVolumeSpecName: "istio-token") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:50.101541 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.101523    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:50.101708 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.101678    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm" (OuterVolumeSpecName: "kube-api-access-w8dsm") pod "9243c8a3-eb3c-49c4-bdc6-906984e7af60" (UID: "9243c8a3-eb3c-49c4-bdc6-906984e7af60"). InnerVolumeSpecName "kube-api-access-w8dsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:50.200920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.200892    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.200920 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.200917    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8dsm\" (UniqueName: \"kubernetes.io/projected/9243c8a3-eb3c-49c4-bdc6-906984e7af60-kube-api-access-w8dsm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.201082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.200927    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.201082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.200936    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.201082 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.200943    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9243c8a3-eb3c-49c4-bdc6-906984e7af60-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:50.740054 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.740018    2564 generic.go:356] "Generic (PLEG): container finished" podID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerID="0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c" exitCode=0
Sep 17 15:56:50.740220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.740077    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"
Sep 17 15:56:50.740220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.740087    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerDied","Data":"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"}
Sep 17 15:56:50.740220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.740136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d" event={"ID":"9243c8a3-eb3c-49c4-bdc6-906984e7af60","Type":"ContainerDied","Data":"9a422c0be0c39fa3b713279f6107907374a9a1ab54a70e0d764fd67520a505f3"}
Sep 17 15:56:50.740220 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.740158    2564 scope.go:117] "RemoveContainer" containerID="0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"
Sep 17 15:56:50.749145 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.749126    2564 scope.go:117] "RemoveContainer" containerID="c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"
Sep 17 15:56:50.756700 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.756679    2564 scope.go:117] "RemoveContainer" containerID="0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"
Sep 17 15:56:50.756965 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:50.756945    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c\": container with ID starting with 0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c not found: ID does not exist" containerID="0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"
Sep 17 15:56:50.757013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.756977    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c"} err="failed to get container status \"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c\": rpc error: code = NotFound desc = could not find container \"0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c\": container with ID starting with 0c1a42efbdfe35a7a4cae86f6c6f26832c76664bde289f79ea047b7e4aef760c not found: ID does not exist"
Sep 17 15:56:50.757013 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.756996    2564 scope.go:117] "RemoveContainer" containerID="c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"
Sep 17 15:56:50.757251 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:50.757231    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17\": container with ID starting with c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17 not found: ID does not exist" containerID="c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"
Sep 17 15:56:50.757307 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.757256    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17"} err="failed to get container status \"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17\": rpc error: code = NotFound desc = could not find container \"c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17\": container with ID starting with c86062165862a9f075825fa7511d2f82241db82d6622e2e93e62cb7ddaeccd17 not found: ID does not exist"
Sep 17 15:56:50.766229 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.766208    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:56:50.773293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:50.773271    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-2b23c7-openshift-default-9784d7d96-zdq2d"]
Sep 17 15:56:52.525989 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:52.525959    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" path="/var/lib/kubelet/pods/9243c8a3-eb3c-49c4-bdc6-906984e7af60/volumes"
Sep 17 15:56:55.961222 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:55.961189    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:56:55.961679 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:55.961407    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" podUID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" containerName="payload-processing" containerID="cri-o://3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7" gracePeriod=30
Sep 17 15:56:55.977715 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:55.977686    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:56:55.977945 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:55.977920    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" podUID="057447e2-1ea6-413d-b791-6a2e43ead46b" containerName="payload-pre-processing" containerID="cri-o://f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce" gracePeriod=30
Sep 17 15:56:56.263814 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.263791    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:56:56.266487 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.266468    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:56:56.456724 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.456671    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume\") pod \"057447e2-1ea6-413d-b791-6a2e43ead46b\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") "
Sep 17 15:56:56.456897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.456750    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5r48\" (UniqueName: \"kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48\") pod \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") "
Sep 17 15:56:56.456897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.456815    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf85z\" (UniqueName: \"kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z\") pod \"057447e2-1ea6-413d-b791-6a2e43ead46b\" (UID: \"057447e2-1ea6-413d-b791-6a2e43ead46b\") "
Sep 17 15:56:56.456897 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.456847    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume\") pod \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\" (UID: \"5dfd4851-b23f-4d47-ae00-bda776ff88c0\") "
Sep 17 15:56:56.457237 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.457202    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "057447e2-1ea6-413d-b791-6a2e43ead46b" (UID: "057447e2-1ea6-413d-b791-6a2e43ead46b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:56.457335 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.457284    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5dfd4851-b23f-4d47-ae00-bda776ff88c0" (UID: "5dfd4851-b23f-4d47-ae00-bda776ff88c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:56.458941 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.458921    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z" (OuterVolumeSpecName: "kube-api-access-tf85z") pod "057447e2-1ea6-413d-b791-6a2e43ead46b" (UID: "057447e2-1ea6-413d-b791-6a2e43ead46b"). InnerVolumeSpecName "kube-api-access-tf85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:56.459015 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.458947    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48" (OuterVolumeSpecName: "kube-api-access-m5r48") pod "5dfd4851-b23f-4d47-ae00-bda776ff88c0" (UID: "5dfd4851-b23f-4d47-ae00-bda776ff88c0"). InnerVolumeSpecName "kube-api-access-m5r48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:56.557808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.557780    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5r48\" (UniqueName: \"kubernetes.io/projected/5dfd4851-b23f-4d47-ae00-bda776ff88c0-kube-api-access-m5r48\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.557808 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.557806    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf85z\" (UniqueName: \"kubernetes.io/projected/057447e2-1ea6-413d-b791-6a2e43ead46b-kube-api-access-tf85z\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.557978 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.557816    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5dfd4851-b23f-4d47-ae00-bda776ff88c0-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.557978 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.557825    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057447e2-1ea6-413d-b791-6a2e43ead46b-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:56.778267 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.778231    2564 generic.go:356] "Generic (PLEG): container finished" podID="057447e2-1ea6-413d-b791-6a2e43ead46b" containerID="f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce" exitCode=0
Sep 17 15:56:56.778453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.778287    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"
Sep 17 15:56:56.778453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.778305    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" event={"ID":"057447e2-1ea6-413d-b791-6a2e43ead46b","Type":"ContainerDied","Data":"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"}
Sep 17 15:56:56.778453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.778381    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78" event={"ID":"057447e2-1ea6-413d-b791-6a2e43ead46b","Type":"ContainerDied","Data":"73ef774cad74347f78eb5f9cf67142e49cec13764f2876ea6e920cf56d4188bb"}
Sep 17 15:56:56.778453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.778405    2564 scope.go:117] "RemoveContainer" containerID="f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"
Sep 17 15:56:56.779744 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.779720    2564 generic.go:356] "Generic (PLEG): container finished" podID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" containerID="3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7" exitCode=0
Sep 17 15:56:56.779862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.779772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"
Sep 17 15:56:56.779862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.779795    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" event={"ID":"5dfd4851-b23f-4d47-ae00-bda776ff88c0","Type":"ContainerDied","Data":"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"}
Sep 17 15:56:56.779862 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.779830    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg" event={"ID":"5dfd4851-b23f-4d47-ae00-bda776ff88c0","Type":"ContainerDied","Data":"3f2a8a16274645f576954b29cd96642a8e039981e8e85be3f4d66c3e9a80694d"}
Sep 17 15:56:56.787142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.787101    2564 scope.go:117] "RemoveContainer" containerID="f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"
Sep 17 15:56:56.787398 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:56.787378    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce\": container with ID starting with f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce not found: ID does not exist" containerID="f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"
Sep 17 15:56:56.787453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.787407    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce"} err="failed to get container status \"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce\": rpc error: code = NotFound desc = could not find container \"f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce\": container with ID starting with f9eada71b5dcab8380602a3d41198202eedd703d90850666f230840c84b1a2ce not found: ID does not exist"
Sep 17 15:56:56.787453 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.787424    2564 scope.go:117] "RemoveContainer" containerID="3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"
Sep 17 15:56:56.795828 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.795801    2564 scope.go:117] "RemoveContainer" containerID="3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"
Sep 17 15:56:56.796209 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:56:56.796179    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7\": container with ID starting with 3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7 not found: ID does not exist" containerID="3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"
Sep 17 15:56:56.796330 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.796217    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7"} err="failed to get container status \"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7\": rpc error: code = NotFound desc = could not find container \"3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7\": container with ID starting with 3f888dbde7077dfe6f58af04cb0f7cf525343ab91a284f4b63c5d70f7cc630e7 not found: ID does not exist"
Sep 17 15:56:56.797764 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.797743    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:56:56.804761 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.804738    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-608b97-55586b7455-7v2wg"]
Sep 17 15:56:56.819390 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.819359    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:56:56.822800 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:56.822773    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-608b97-5c94c45554-dxr78"]
Sep 17 15:56:57.622545 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.622500    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" podUID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" containerName="main" containerID="cri-o://640909a5bd1901bc1794bd48913242d77e915bafa298207d77ce1846e6b77752" gracePeriod=45
Sep 17 15:56:57.790112 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.789975    2564 generic.go:356] "Generic (PLEG): container finished" podID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" containerID="640909a5bd1901bc1794bd48913242d77e915bafa298207d77ce1846e6b77752" exitCode=0
Sep 17 15:56:57.790112 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.790076    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" event={"ID":"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f","Type":"ContainerDied","Data":"640909a5bd1901bc1794bd48913242d77e915bafa298207d77ce1846e6b77752"}
Sep 17 15:56:57.873974 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.873917    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:56:57.975563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975523    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975563 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975564    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975797 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975598    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975797 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975644    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975797 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975719    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975925 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975806    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2jjn\" (UniqueName: \"kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn\") pod \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\" (UID: \"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f\") "
Sep 17 15:56:57.975925 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home" (OuterVolumeSpecName: "home") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:57.975925 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.975877    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:57.976085 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.976061    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache" (OuterVolumeSpecName: "model-cache") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:57.976191 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.976175    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:57.976253 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.976194    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:57.976253 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.976203    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:57.977944 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.977905    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm" (OuterVolumeSpecName: "dshm") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:57.977944 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.977913    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:57.978108 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:57.977997    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn" (OuterVolumeSpecName: "kube-api-access-l2jjn") pod "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" (UID: "4107d15b-fd3c-4915-a9b0-d971f8d0ae7f"). InnerVolumeSpecName "kube-api-access-l2jjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:58.076836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.076799    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2jjn\" (UniqueName: \"kubernetes.io/projected/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-kube-api-access-l2jjn\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:58.076836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.076831    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:58.076836 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.076840    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:56:58.527036 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.526992    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057447e2-1ea6-413d-b791-6a2e43ead46b" path="/var/lib/kubelet/pods/057447e2-1ea6-413d-b791-6a2e43ead46b/volumes"
Sep 17 15:56:58.527438 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.527419    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" path="/var/lib/kubelet/pods/5dfd4851-b23f-4d47-ae00-bda776ff88c0/volumes"
Sep 17 15:56:58.797017 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.796939    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"
Sep 17 15:56:58.797017 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.796977    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm" event={"ID":"4107d15b-fd3c-4915-a9b0-d971f8d0ae7f","Type":"ContainerDied","Data":"5c1069d94131742d84d82c92a830c33469d12e843ba2f9a6a08d7761b7a3b1a3"}
Sep 17 15:56:58.797518 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.797031    2564 scope.go:117] "RemoveContainer" containerID="640909a5bd1901bc1794bd48913242d77e915bafa298207d77ce1846e6b77752"
Sep 17 15:56:58.829491 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.829455    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:56:58.839843 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:58.839816    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/auth-test-model-5818c9-kserve-f46cf9cdc-8f4nm"]
Sep 17 15:56:59.983571 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:56:59.983523    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:57:00.524903 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:00.524867    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" path="/var/lib/kubelet/pods/4107d15b-fd3c-4915-a9b0-d971f8d0ae7f/volumes"
Sep 17 15:57:36.761142 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761108    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:57:36.761635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761571    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:57:36.761635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761587    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:57:36.761635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761599    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" containerName="payload-processing"
Sep 17 15:57:36.761635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761606    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" containerName="payload-processing"
Sep 17 15:57:36.761635 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761633    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761745    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" containerName="main"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761757    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" containerName="main"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761780    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" containerName="payload-processing"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761786    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" containerName="payload-processing"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761795    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="057447e2-1ea6-413d-b791-6a2e43ead46b" containerName="payload-pre-processing"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761801    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="057447e2-1ea6-413d-b791-6a2e43ead46b" containerName="payload-pre-processing"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761822    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761832    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761925    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761935    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761942    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9243c8a3-eb3c-49c4-bdc6-906984e7af60" containerName="istio-proxy"
Sep 17 15:57:36.761942 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761947    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4107d15b-fd3c-4915-a9b0-d971f8d0ae7f" containerName="main"
Sep 17 15:57:36.762381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761955    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="057447e2-1ea6-413d-b791-6a2e43ead46b" containerName="payload-pre-processing"
Sep 17 15:57:36.762381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761961    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="11685c29-4640-442d-a39c-94291e185ea7" containerName="istio-proxy"
Sep 17 15:57:36.762381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761966    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfd4851-b23f-4d47-ae00-bda776ff88c0" containerName="payload-processing"
Sep 17 15:57:36.762381 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.761972    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a726c89e-61ff-4e83-8ecc-6fb8b13bdfba" containerName="payload-processing"
Sep 17 15:57:36.763905 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.763887    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.766293 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.766267    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"kube-root-ca.crt\""
Sep 17 15:57:36.766416 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.766273    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"openshift-service-ca.crt\""
Sep 17 15:57:36.767092 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.767065    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"default-dockercfg-dxwws\""
Sep 17 15:57:36.767188 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.767110    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-5818c9\"/\"sub-test-model-5818c9-kserve-self-signed-certs\""
Sep 17 15:57:36.776931 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.776908    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:57:36.837492 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837460    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.837683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837521    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.837683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.837683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837604    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.837683 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837672    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bbjr\" (UniqueName: \"kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.837895 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.837715    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938718 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938672    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938742    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938767    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938799    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bbjr\" (UniqueName: \"kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938841    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.938969 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.938953    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.939261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.939064    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.939261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.939085    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.939261 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.939246    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.941073 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.941049    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.941490 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.941469    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:36.949107 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:36.949086    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bbjr\" (UniqueName: \"kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr\") pod \"sub-test-model-5818c9-kserve-75f448544c-cfk9s\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") " pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:37.074062 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:37.073978    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:37.347226 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:37.347202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:57:37.349189 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:57:37.349165    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1305fe0c_a6f7_47a3_aca4_b0584ccf1f13.slice/crio-9b7481e29a82b2fc13767859c449aa4fee50852219f2344ab0a213b8df873280 WatchSource:0}: Error finding container 9b7481e29a82b2fc13767859c449aa4fee50852219f2344ab0a213b8df873280: Status 404 returned error can't find the container with id 9b7481e29a82b2fc13767859c449aa4fee50852219f2344ab0a213b8df873280
Sep 17 15:57:38.015849 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:38.015815    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" event={"ID":"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13","Type":"ContainerStarted","Data":"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"}
Sep 17 15:57:38.015849 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:38.015856    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" event={"ID":"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13","Type":"ContainerStarted","Data":"9b7481e29a82b2fc13767859c449aa4fee50852219f2344ab0a213b8df873280"}
Sep 17 15:57:38.034411 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:38.034361    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" podStartSLOduration=2.034345043 podStartE2EDuration="2.034345043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:57:38.033342399 +0000 UTC m=+1650.000520221" watchObservedRunningTime="2026-09-17 15:57:38.034345043 +0000 UTC m=+1650.001522860"
Sep 17 15:57:40.571087 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:40.571038    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:57:47.074389 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:47.074349    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:47.074814 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:47.074404    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:47.086916 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:47.086894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:48.083172 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:48.083141    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:57:55.979115 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:57:55.979079    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:58:09.033483 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:09.033451    2564 scope.go:117] "RemoveContainer" containerID="0ed459c72847adc72c5af23c3102de9a94f7bf68e4167616f85a3c5089ac5a57"
Sep 17 15:58:09.080926 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:09.080898    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:58:17.269000 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:17.268968    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:58:19.986327 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:19.986289    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:58:30.188170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:30.188131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:58:32.287531 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.287482    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" podUID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" containerName="main" containerID="cri-o://f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a" gracePeriod=45
Sep 17 15:58:32.532369 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.532345    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:58:32.674685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674573    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.674685 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674674    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.674921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674698    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.674921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674759    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.674921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674817    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bbjr\" (UniqueName: \"kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.674921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674822    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:32.674921 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.674840    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache\") pod \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\" (UID: \"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13\") "
Sep 17 15:58:32.675165 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.675031    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home" (OuterVolumeSpecName: "home") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:32.675165 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.675130    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache" (OuterVolumeSpecName: "model-cache") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:32.675265 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.675193    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:32.675265 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.675211    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:32.675265 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.675226    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:32.676858 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.676836    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr" (OuterVolumeSpecName: "kube-api-access-2bbjr") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "kube-api-access-2bbjr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:32.676962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.676856    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm" (OuterVolumeSpecName: "dshm") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:32.676962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.676862    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" (UID: "1305fe0c-a6f7-47a3-aca4-b0584ccf1f13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:32.775879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.775850    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bbjr\" (UniqueName: \"kubernetes.io/projected/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-kube-api-access-2bbjr\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:32.775879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.775875    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:32.775879 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:32.775885    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 15:58:33.300859 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.300825    2564 generic.go:356] "Generic (PLEG): container finished" podID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" containerID="f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a" exitCode=0
Sep 17 15:58:33.301236 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.300866    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"
Sep 17 15:58:33.301236 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.300906    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" event={"ID":"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13","Type":"ContainerDied","Data":"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"}
Sep 17 15:58:33.301236 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.300944    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s" event={"ID":"1305fe0c-a6f7-47a3-aca4-b0584ccf1f13","Type":"ContainerDied","Data":"9b7481e29a82b2fc13767859c449aa4fee50852219f2344ab0a213b8df873280"}
Sep 17 15:58:33.301236 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.300963    2564 scope.go:117] "RemoveContainer" containerID="f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"
Sep 17 15:58:33.317863 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.317843    2564 scope.go:117] "RemoveContainer" containerID="f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"
Sep 17 15:58:33.318166 ip-10-0-141-17 kubenswrapper[2564]: E0917 15:58:33.318139    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a\": container with ID starting with f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a not found: ID does not exist" containerID="f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"
Sep 17 15:58:33.318263 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.318176    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a"} err="failed to get container status \"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a\": rpc error: code = NotFound desc = could not find container \"f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a\": container with ID starting with f6cb2b062057727fb1342fa65b41e278292d6957c2e6d5d62b06d8427997b57a not found: ID does not exist"
Sep 17 15:58:33.338331 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.338308    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:58:33.353837 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:33.353811    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-5818c9/sub-test-model-5818c9-kserve-75f448544c-cfk9s"]
Sep 17 15:58:34.526023 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:34.525992    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" path="/var/lib/kubelet/pods/1305fe0c-a6f7-47a3-aca4-b0584ccf1f13/volumes"
Sep 17 15:58:53.891388 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:58:53.891349    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:59:15.483170 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:15.483135    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:59:25.877599 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:25.877566    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:59:39.484217 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484175    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 15:59:39.484661 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484646    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:59:39.484713 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484663    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:59:39.484713 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484684    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" containerName="main"
Sep 17 15:59:39.484713 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484689    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" containerName="main"
Sep 17 15:59:39.484806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484752    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ccedd1d-af00-43f4-8c59-b7f679c36092" containerName="istio-proxy"
Sep 17 15:59:39.484806 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.484766    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1305fe0c-a6f7-47a3-aca4-b0584ccf1f13" containerName="main"
Sep 17 15:59:39.487925 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.487908    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.491865 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.491846    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-420219\""
Sep 17 15:59:39.501962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.501938    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 15:59:39.596089 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.596060    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgw7f\" (UniqueName: \"kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.596257 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.596225    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.629528 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.629489    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 15:59:39.632986 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.632961    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.635354 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.635331    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-420219-dockercfg-tdw7k\""
Sep 17 15:59:39.645816 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.645794    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 15:59:39.697163 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.697132    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgw7f\" (UniqueName: \"kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.697321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.697183    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn4bx\" (UniqueName: \"kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.697321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.697223    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.697321 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.697293    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.697980 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.697962    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.706265 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.706237    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgw7f\" (UniqueName: \"kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f\") pod \"payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.798337 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.798299    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:39.798552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.798526    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn4bx\" (UniqueName: \"kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.798603 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.798572    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.799229 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.799211    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.810768 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.810743    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn4bx\" (UniqueName: \"kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx\") pod \"payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:39.891630 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.891591    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:59:39.945469 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:39.945438    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:40.077223 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.077202    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 15:59:40.079198 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:59:40.079172    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ba772c2_ba7d_405d_87d3_a0d9604bd4dd.slice/crio-3b6882c911679819c55050112b18820ff97048bd5ed31f43463934c8c165c74e WatchSource:0}: Error finding container 3b6882c911679819c55050112b18820ff97048bd5ed31f43463934c8c165c74e: Status 404 returned error can't find the container with id 3b6882c911679819c55050112b18820ff97048bd5ed31f43463934c8c165c74e
Sep 17 15:59:40.227226 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.227190    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 15:59:40.230095 ip-10-0-141-17 kubenswrapper[2564]: W0917 15:59:40.230067    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5131578a_4b62_4d1e_aa71_3f61125ed3cb.slice/crio-0ce3d14d549cfd0d95f5e7c2da96bd57452dc8c84d38e1a8c5e54acf87138c88 WatchSource:0}: Error finding container 0ce3d14d549cfd0d95f5e7c2da96bd57452dc8c84d38e1a8c5e54acf87138c88: Status 404 returned error can't find the container with id 0ce3d14d549cfd0d95f5e7c2da96bd57452dc8c84d38e1a8c5e54acf87138c88
Sep 17 15:59:40.659983 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.659948    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" event={"ID":"5131578a-4b62-4d1e-aa71-3f61125ed3cb","Type":"ContainerStarted","Data":"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"}
Sep 17 15:59:40.660417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.660002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" event={"ID":"5131578a-4b62-4d1e-aa71-3f61125ed3cb","Type":"ContainerStarted","Data":"0ce3d14d549cfd0d95f5e7c2da96bd57452dc8c84d38e1a8c5e54acf87138c88"}
Sep 17 15:59:40.660417 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.660125    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 15:59:40.661566 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.661539    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" event={"ID":"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd","Type":"ContainerStarted","Data":"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"}
Sep 17 15:59:40.661712 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.661582    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" event={"ID":"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd","Type":"ContainerStarted","Data":"3b6882c911679819c55050112b18820ff97048bd5ed31f43463934c8c165c74e"}
Sep 17 15:59:40.661712 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.661660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:40.681962 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.681913    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" podStartSLOduration=1.6819000929999999 podStartE2EDuration="1.681900093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:40.68086371 +0000 UTC m=+1772.648041529" watchObservedRunningTime="2026-09-17 15:59:40.681900093 +0000 UTC m=+1772.649077953"
Sep 17 15:59:40.700391 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:40.700351    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" podStartSLOduration=1.7003389260000001 podStartE2EDuration="1.700338926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:40.699421662 +0000 UTC m=+1772.666599492" watchObservedRunningTime="2026-09-17 15:59:40.700338926 +0000 UTC m=+1772.667516743"
Sep 17 15:59:48.595256 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:48.595217    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 15:59:51.668123 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:51.668086    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 15:59:51.668552 ip-10-0-141-17 kubenswrapper[2564]: I0917 15:59:51.668169    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 16:00:00.144013 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.143979    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"]
Sep 17 16:00:00.147715 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.147692    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"]
Sep 17 16:00:00.147877 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.147827    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"
Sep 17 16:00:00.151164 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.151144    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"]
Sep 17 16:00:00.151294 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.151263    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"
Sep 17 16:00:00.152689 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.152673    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 16:00:00.154311 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.154294    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"
Sep 17 16:00:00.159381 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.159365    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"
Sep 17 16:00:00.165602 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.165575    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"]
Sep 17 16:00:00.173860 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.173765    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"]
Sep 17 16:00:00.174407 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.174387    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"
Sep 17 16:00:00.194394 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.194210    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"]
Sep 17 16:00:00.211813 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.211636    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"
Sep 17 16:00:00.473274 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.473248    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"]
Sep 17 16:00:00.477940 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:00:00.477832    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac8e5d8_5911_4d53_b73d_69ae0cbce654.slice/crio-cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005 WatchSource:0}: Error finding container cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005: Status 404 returned error can't find the container with id cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005
Sep 17 16:00:00.510134 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.510112    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"]
Sep 17 16:00:00.511718 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:00:00.511695    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633ed79b_ee89_45f6_9b68_adda1e74d51f.slice/crio-ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430 WatchSource:0}: Error finding container ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430: Status 404 returned error can't find the container with id ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430
Sep 17 16:00:00.565121 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.565096    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"]
Sep 17 16:00:00.567057 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:00:00.567033    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae690e55_e993_4ce0_b404_45e9f734b2d0.slice/crio-3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9 WatchSource:0}: Error finding container 3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9: Status 404 returned error can't find the container with id 3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9
Sep 17 16:00:00.765633 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.765576    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp" event={"ID":"ae690e55-e993-4ce0-b404-45e9f734b2d0","Type":"ContainerStarted","Data":"3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9"}
Sep 17 16:00:00.766868 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.766845    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9" event={"ID":"633ed79b-ee89-45f6-9b68-adda1e74d51f","Type":"ContainerStarted","Data":"ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430"}
Sep 17 16:00:00.768053 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:00.768032    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27" event={"ID":"fac8e5d8-5911-4d53-b73d-69ae0cbce654","Type":"ContainerStarted","Data":"cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005"}
Sep 17 16:00:01.775263 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.775226    2564 generic.go:356] "Generic (PLEG): container finished" podID="633ed79b-ee89-45f6-9b68-adda1e74d51f" containerID="fbf240945098cb133d4238f11148ab666210571dee5d57e2f310c52ffe3a74ee" exitCode=0
Sep 17 16:00:01.775671 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.775313    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9" event={"ID":"633ed79b-ee89-45f6-9b68-adda1e74d51f","Type":"ContainerDied","Data":"fbf240945098cb133d4238f11148ab666210571dee5d57e2f310c52ffe3a74ee"}
Sep 17 16:00:01.776834 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.776808    2564 generic.go:356] "Generic (PLEG): container finished" podID="fac8e5d8-5911-4d53-b73d-69ae0cbce654" containerID="3cdc360b5bba5b2222b7b57dfccaca11100211bd856f53e4febb2f9f1b207b82" exitCode=0
Sep 17 16:00:01.776940 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.776865    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27" event={"ID":"fac8e5d8-5911-4d53-b73d-69ae0cbce654","Type":"ContainerDied","Data":"3cdc360b5bba5b2222b7b57dfccaca11100211bd856f53e4febb2f9f1b207b82"}
Sep 17 16:00:01.778313 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.778295    2564 generic.go:356] "Generic (PLEG): container finished" podID="ae690e55-e993-4ce0-b404-45e9f734b2d0" containerID="c3e3feeafcecec39d1631bec22a92f19223e2027ddf1beea38e6ab9b6415d486" exitCode=0
Sep 17 16:00:01.778402 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:01.778319    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp" event={"ID":"ae690e55-e993-4ce0-b404-45e9f734b2d0","Type":"ContainerDied","Data":"c3e3feeafcecec39d1631bec22a92f19223e2027ddf1beea38e6ab9b6415d486"}
Sep 17 16:00:02.925576 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:02.925543    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"
Sep 17 16:00:02.973442 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:02.973417    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"
Sep 17 16:00:02.976544 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:02.976525    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"
Sep 17 16:00:03.790489 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.790447    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9" event={"ID":"633ed79b-ee89-45f6-9b68-adda1e74d51f","Type":"ContainerDied","Data":"ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430"}
Sep 17 16:00:03.790686 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.790513    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6dc937e5330adf293f227271f661220de2ff64064219197fbac1d7d6fa1430"
Sep 17 16:00:03.790686 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.790458    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827680-jsks9"
Sep 17 16:00:03.791801 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.791781    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"
Sep 17 16:00:03.791906 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.791810    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27" event={"ID":"fac8e5d8-5911-4d53-b73d-69ae0cbce654","Type":"ContainerDied","Data":"cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005"}
Sep 17 16:00:03.791906 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.791856    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf9cb0333cecafc6b524acdb664f2f857d53d534dc884d9424beaa18cd56b005"
Sep 17 16:00:03.793381 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.793360    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"
Sep 17 16:00:03.793465 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.793367    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp" event={"ID":"ae690e55-e993-4ce0-b404-45e9f734b2d0","Type":"ContainerDied","Data":"3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9"}
Sep 17 16:00:03.793518 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:03.793475    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d9c7c574942acbbac768f60adefff5dfd522799c64c3bc833b64541f409fdc9"
Sep 17 16:00:08.964548 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.964515    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:00:08.965129 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965108    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac8e5d8-5911-4d53-b73d-69ae0cbce654" containerName="cleanup"
Sep 17 16:00:08.965230 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965133    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac8e5d8-5911-4d53-b73d-69ae0cbce654" containerName="cleanup"
Sep 17 16:00:08.965230 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965145    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae690e55-e993-4ce0-b404-45e9f734b2d0" containerName="cleanup"
Sep 17 16:00:08.965230 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965152    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae690e55-e993-4ce0-b404-45e9f734b2d0" containerName="cleanup"
Sep 17 16:00:08.965230 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965166    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="633ed79b-ee89-45f6-9b68-adda1e74d51f" containerName="cleanup"
Sep 17 16:00:08.965230 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965171    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="633ed79b-ee89-45f6-9b68-adda1e74d51f" containerName="cleanup"
Sep 17 16:00:08.965438 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965244    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac8e5d8-5911-4d53-b73d-69ae0cbce654" containerName="cleanup"
Sep 17 16:00:08.965438 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965255    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae690e55-e993-4ce0-b404-45e9f734b2d0" containerName="cleanup"
Sep 17 16:00:08.965438 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.965265    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="633ed79b-ee89-45f6-9b68-adda1e74d51f" containerName="cleanup"
Sep 17 16:00:08.968379 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.968360    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:08.971050 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.971025    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ddedf5\""
Sep 17 16:00:08.979416 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.979396    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ddedf5-dockercfg-9h2cq\""
Sep 17 16:00:08.988405 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:08.988379    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:00:09.082572 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.082538    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhtpk\" (UniqueName: \"kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.082572 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.082581    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.184229 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.184191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhtpk\" (UniqueName: \"kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.184229 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.184230    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.184889 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.184869    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.197743 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.197712    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhtpk\" (UniqueName: \"kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk\") pod \"payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.279825 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.279786    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.569046 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.569021    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:00:09.570397 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:00:09.570376    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb5d730_abcd_4d4d_acab_d6fdad40b28a.slice/crio-c646297b6b8933fc856f5d8495dec8cbdfe329ed0847cbefe1b782ebea513c04 WatchSource:0}: Error finding container c646297b6b8933fc856f5d8495dec8cbdfe329ed0847cbefe1b782ebea513c04: Status 404 returned error can't find the container with id c646297b6b8933fc856f5d8495dec8cbdfe329ed0847cbefe1b782ebea513c04
Sep 17 16:00:09.828791 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.828691    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" event={"ID":"adb5d730-abcd-4d4d-acab-d6fdad40b28a","Type":"ContainerStarted","Data":"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"}
Sep 17 16:00:09.828791 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.828748    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" event={"ID":"adb5d730-abcd-4d4d-acab-d6fdad40b28a","Type":"ContainerStarted","Data":"c646297b6b8933fc856f5d8495dec8cbdfe329ed0847cbefe1b782ebea513c04"}
Sep 17 16:00:09.829013 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.828894    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:09.857039 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:09.856982    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" podStartSLOduration=1.856966163 podStartE2EDuration="1.856966163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:09.854679334 +0000 UTC m=+1801.821857153" watchObservedRunningTime="2026-09-17 16:00:09.856966163 +0000 UTC m=+1801.824143972"
Sep 17 16:00:12.282935 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:12.282897    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:00:20.837680 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:20.837648    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:00:50.090104 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.090067    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:00:50.093590 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.093570    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.097024 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.096993    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-420219\"/\"openshift-service-ca.crt\""
Sep 17 16:00:50.097024 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.097015    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-420219\"/\"kube-root-ca.crt\""
Sep 17 16:00:50.097174 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.097020    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-420219\"/\"ipp-route-420219-kserve-self-signed-certs\""
Sep 17 16:00:50.097174 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.097157    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-420219\"/\"default-dockercfg-l6pwq\""
Sep 17 16:00:50.111914 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.111891    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:00:50.194556 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194518    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.194732 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194563    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.194732 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194594    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwt24\" (UniqueName: \"kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.194732 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194689    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.194732 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194724    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.194879 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.194743    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295527 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295495    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295527 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295531    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295782 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295554    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwt24\" (UniqueName: \"kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295782 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295595    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295782 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295644    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.295782 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295667    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.296005 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.295984    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.296056 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.296002    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.296094 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.296051    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.297954 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.297930    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.298095 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.298080    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.308049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.308020    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwt24\" (UniqueName: \"kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24\") pod \"ipp-route-420219-kserve-86b64645bc-jjmhs\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") " pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.403286 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.403203    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:00:50.692689 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:50.692596    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:00:50.696759 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:00:50.696730    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0cf28817_ef1a_4401_a215_387d1b72fd94.slice/crio-f88da42268dfc6287cf438284bfe3c0080ebb6b5ffb6fcf6c9f3ff7bc5ef0016 WatchSource:0}: Error finding container f88da42268dfc6287cf438284bfe3c0080ebb6b5ffb6fcf6c9f3ff7bc5ef0016: Status 404 returned error can't find the container with id f88da42268dfc6287cf438284bfe3c0080ebb6b5ffb6fcf6c9f3ff7bc5ef0016
Sep 17 16:00:51.060919 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:51.060873    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" event={"ID":"0cf28817-ef1a-4401-a215-387d1b72fd94","Type":"ContainerStarted","Data":"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"}
Sep 17 16:00:51.061097 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:51.060930    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" event={"ID":"0cf28817-ef1a-4401-a215-387d1b72fd94","Type":"ContainerStarted","Data":"f88da42268dfc6287cf438284bfe3c0080ebb6b5ffb6fcf6c9f3ff7bc5ef0016"}
Sep 17 16:00:51.092752 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:51.092703    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" podStartSLOduration=1.092688293 podStartE2EDuration="1.092688293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:00:51.089258549 +0000 UTC m=+1843.056436367" watchObservedRunningTime="2026-09-17 16:00:51.092688293 +0000 UTC m=+1843.059866111"
Sep 17 16:00:53.797967 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:00:53.797926    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:01:00.403578 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:00.403537    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:01:00.403965 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:00.403600    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:01:00.416456 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:00.416429    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:01:01.127289 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:01.127251    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:01:07.197747 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:07.197711    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:01:09.076012 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:09.075979    2564 scope.go:117] "RemoveContainer" containerID="c66bed2ccc86c68cf2c55e6db5a47b6863093bbe806a67beb90e3f51d56eca79"
Sep 17 16:01:09.084357 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:09.084333    2564 scope.go:117] "RemoveContainer" containerID="9790abc0efbcdbfe7b8e8f3721821447c4cab29eccd7443737162f8283bbca99"
Sep 17 16:01:12.488241 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:12.488211    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4_5131578a-4b62-4d1e-aa71-3f61125ed3cb/payload-processing/0.log"
Sep 17 16:01:23.117567 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.117533    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:01:23.121572 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.121550    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.123926 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.123901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-ec94ed\""
Sep 17 16:01:23.124172 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.124156    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-ec94ed-dockercfg-trq72\""
Sep 17 16:01:23.133525 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.133503    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:01:23.217835 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.217797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.217992 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.217881    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hlk4\" (UniqueName: \"kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.318757 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.318724    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hlk4\" (UniqueName: \"kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.318927 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.318836    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.319524 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.319500    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.328244 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.328221    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hlk4\" (UniqueName: \"kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4\") pod \"payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.433424 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.433347    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:23.707388 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:23.707362    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:01:23.712978 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:01:23.712944    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b7ee59_0baf_4b2c_a2fe_e8734995b2a8.slice/crio-7d1fe2740e94e7e5592e3b4e86b7cf30fb1873953593b43d8532ad8d6732ef89 WatchSource:0}: Error finding container 7d1fe2740e94e7e5592e3b4e86b7cf30fb1873953593b43d8532ad8d6732ef89: Status 404 returned error can't find the container with id 7d1fe2740e94e7e5592e3b4e86b7cf30fb1873953593b43d8532ad8d6732ef89
Sep 17 16:01:24.257115 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:24.257083    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" event={"ID":"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8","Type":"ContainerStarted","Data":"7469e02f8f4f075e257ae0357862c4779ef7f31d2520882aae17d3fd13238953"}
Sep 17 16:01:24.257557 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:24.257136    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" event={"ID":"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8","Type":"ContainerStarted","Data":"7d1fe2740e94e7e5592e3b4e86b7cf30fb1873953593b43d8532ad8d6732ef89"}
Sep 17 16:01:24.257557 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:24.257217    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:24.317510 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:24.317461    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" podStartSLOduration=1.317448425 podStartE2EDuration="1.317448425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:24.31554742 +0000 UTC m=+1876.282725241" watchObservedRunningTime="2026-09-17 16:01:24.317448425 +0000 UTC m=+1876.284626267"
Sep 17 16:01:27.566817 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:27.566782    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:01:35.264520 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:35.264488    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:47.668070 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.668030    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"]
Sep 17 16:01:47.672196 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.672176    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:47.674856 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.674836    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 16:01:47.675364 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.675346    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 16:01:47.688858 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.688833    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"]
Sep 17 16:01:47.738758 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.738721    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f\" (UID: \"96d42ca9-d5ae-49aa-a746-3da915e13009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:47.839589 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.839546    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f\" (UID: \"96d42ca9-d5ae-49aa-a746-3da915e13009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:47.840157 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.840137    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f\" (UID: \"96d42ca9-d5ae-49aa-a746-3da915e13009\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:47.982329 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:47.982257    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:48.253473 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:48.253451    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"]
Sep 17 16:01:48.255313 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:01:48.255283    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d42ca9_d5ae_49aa_a746_3da915e13009.slice/crio-f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c WatchSource:0}: Error finding container f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c: Status 404 returned error can't find the container with id f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c
Sep 17 16:01:48.386872 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:48.386843    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f" event={"ID":"96d42ca9-d5ae-49aa-a746-3da915e13009","Type":"ContainerStarted","Data":"f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c"}
Sep 17 16:01:49.394103 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:49.394062    2564 generic.go:356] "Generic (PLEG): container finished" podID="96d42ca9-d5ae-49aa-a746-3da915e13009" containerID="74dd965608f4cc11e77b86d91cd83e35cff414fbff30595828f7bdb6a8b09e18" exitCode=0
Sep 17 16:01:49.394546 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:49.394153    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f" event={"ID":"96d42ca9-d5ae-49aa-a746-3da915e13009","Type":"ContainerDied","Data":"74dd965608f4cc11e77b86d91cd83e35cff414fbff30595828f7bdb6a8b09e18"}
Sep 17 16:01:50.543364 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:50.543340    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:50.567129 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:50.567105    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca\") pod \"96d42ca9-d5ae-49aa-a746-3da915e13009\" (UID: \"96d42ca9-d5ae-49aa-a746-3da915e13009\") "
Sep 17 16:01:50.567440 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:50.567414    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96d42ca9-d5ae-49aa-a746-3da915e13009" (UID: "96d42ca9-d5ae-49aa-a746-3da915e13009"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:50.668486 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:50.668421    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d42ca9-d5ae-49aa-a746-3da915e13009-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:01:51.408785 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:51.408755    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"
Sep 17 16:01:51.408951 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:51.408791    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f" event={"ID":"96d42ca9-d5ae-49aa-a746-3da915e13009","Type":"ContainerDied","Data":"f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c"}
Sep 17 16:01:51.408951 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:51.408836    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1449985e939a7e223424ea49d8ca46a1761a1c9bdcabee892fe500ec3d8193c"
Sep 17 16:01:59.335062 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.335031    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:01:59.335550 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.335274    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" podUID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" containerName="payload-processing" containerID="cri-o://7469e02f8f4f075e257ae0357862c4779ef7f31d2520882aae17d3fd13238953" gracePeriod=30
Sep 17 16:01:59.453672 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.453645    2564 generic.go:356] "Generic (PLEG): container finished" podID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" containerID="7469e02f8f4f075e257ae0357862c4779ef7f31d2520882aae17d3fd13238953" exitCode=0
Sep 17 16:01:59.453774 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.453719    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" event={"ID":"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8","Type":"ContainerDied","Data":"7469e02f8f4f075e257ae0357862c4779ef7f31d2520882aae17d3fd13238953"}
Sep 17 16:01:59.593905 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.593842    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:01:59.654391 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.654354    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume\") pod \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") "
Sep 17 16:01:59.654739 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.654708    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" (UID: "67b7ee59-0baf-4b2c-a2fe-e8734995b2a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:59.654739 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.654722    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hlk4\" (UniqueName: \"kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4\") pod \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\" (UID: \"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8\") "
Sep 17 16:01:59.655125 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.655104    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:01:59.656858 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.656830    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4" (OuterVolumeSpecName: "kube-api-access-4hlk4") pod "67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" (UID: "67b7ee59-0baf-4b2c-a2fe-e8734995b2a8"). InnerVolumeSpecName "kube-api-access-4hlk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:59.719217 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.719194    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"]
Sep 17 16:01:59.722882 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.722862    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-ec94ed-rgx8f"]
Sep 17 16:01:59.757019 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:01:59.756595    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hlk4\" (UniqueName: \"kubernetes.io/projected/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8-kube-api-access-4hlk4\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:00.459995 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.459967    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"
Sep 17 16:02:00.460410 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.460004    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5" event={"ID":"67b7ee59-0baf-4b2c-a2fe-e8734995b2a8","Type":"ContainerDied","Data":"7d1fe2740e94e7e5592e3b4e86b7cf30fb1873953593b43d8532ad8d6732ef89"}
Sep 17 16:02:00.460410 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.460064    2564 scope.go:117] "RemoveContainer" containerID="7469e02f8f4f075e257ae0357862c4779ef7f31d2520882aae17d3fd13238953"
Sep 17 16:02:00.510918 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.510895    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:02:00.525667 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.525643    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d42ca9-d5ae-49aa-a746-3da915e13009" path="/var/lib/kubelet/pods/96d42ca9-d5ae-49aa-a746-3da915e13009/volumes"
Sep 17 16:02:00.525918 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:00.525906    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-ec94ed-7997898fb7-67jf5"]
Sep 17 16:02:02.524975 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:02.524942    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" path="/var/lib/kubelet/pods/67b7ee59-0baf-4b2c-a2fe-e8734995b2a8/volumes"
Sep 17 16:02:04.012421 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:04.012386    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:02:10.829486 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.829451    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"]
Sep 17 16:02:10.829916 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.829901    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d42ca9-d5ae-49aa-a746-3da915e13009" containerName="revoke-keys"
Sep 17 16:02:10.829961 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.829918    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d42ca9-d5ae-49aa-a746-3da915e13009" containerName="revoke-keys"
Sep 17 16:02:10.829961 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.829932    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" containerName="payload-processing"
Sep 17 16:02:10.829961 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.829938    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" containerName="payload-processing"
Sep 17 16:02:10.830056 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.830008    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d42ca9-d5ae-49aa-a746-3da915e13009" containerName="revoke-keys"
Sep 17 16:02:10.830056 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.830017    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b7ee59-0baf-4b2c-a2fe-e8734995b2a8" containerName="payload-processing"
Sep 17 16:02:10.834879 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.834859    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:10.838794 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.838773    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 16:02:10.839009 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.838994    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 16:02:10.843462 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.843440    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"]
Sep 17 16:02:10.969851 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:10.969824    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj\" (UID: \"a0611e31-f0c8-41d3-9bd9-50c351bea08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:11.070527 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:11.070494    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj\" (UID: \"a0611e31-f0c8-41d3-9bd9-50c351bea08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:11.071177 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:11.071157    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj\" (UID: \"a0611e31-f0c8-41d3-9bd9-50c351bea08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:11.146388 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:11.146300    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:11.418990 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:11.418965    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"]
Sep 17 16:02:11.420632 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:02:11.420590    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0611e31_f0c8_41d3_9bd9_50c351bea08c.slice/crio-9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a WatchSource:0}: Error finding container 9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a: Status 404 returned error can't find the container with id 9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a
Sep 17 16:02:11.522046 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:11.522019    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj" event={"ID":"a0611e31-f0c8-41d3-9bd9-50c351bea08c","Type":"ContainerStarted","Data":"9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a"}
Sep 17 16:02:12.530717 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:12.530678    2564 generic.go:356] "Generic (PLEG): container finished" podID="a0611e31-f0c8-41d3-9bd9-50c351bea08c" containerID="01a8599304219502659abda6903c7055a8f89dfde6709ee069c10e67eea6a12e" exitCode=0
Sep 17 16:02:12.531095 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:12.530751    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj" event={"ID":"a0611e31-f0c8-41d3-9bd9-50c351bea08c","Type":"ContainerDied","Data":"01a8599304219502659abda6903c7055a8f89dfde6709ee069c10e67eea6a12e"}
Sep 17 16:02:13.676097 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:13.676077    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:13.795897 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:13.795820    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca\") pod \"a0611e31-f0c8-41d3-9bd9-50c351bea08c\" (UID: \"a0611e31-f0c8-41d3-9bd9-50c351bea08c\") "
Sep 17 16:02:13.796234 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:13.796209    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0611e31-f0c8-41d3-9bd9-50c351bea08c" (UID: "a0611e31-f0c8-41d3-9bd9-50c351bea08c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:13.897017 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:13.896986    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0611e31-f0c8-41d3-9bd9-50c351bea08c-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:14.543204 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:14.543172    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"
Sep 17 16:02:14.543346 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:14.543206    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj" event={"ID":"a0611e31-f0c8-41d3-9bd9-50c351bea08c","Type":"ContainerDied","Data":"9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a"}
Sep 17 16:02:14.543346 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:14.543262    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9677cc7d7c7026198ba313d384a16430094ea6f7ec04f03eae87ba8dc2f8ea9a"
Sep 17 16:02:24.247609 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:24.247571    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:02:26.846299 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:26.846264    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 16:02:26.846785 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:26.846524    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" podUID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" containerName="payload-processing" containerID="cri-o://04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7" gracePeriod=30
Sep 17 16:02:26.872819 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:26.872793    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 16:02:26.872999 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:26.872979    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" podUID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" containerName="payload-pre-processing" containerID="cri-o://af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a" gracePeriod=30
Sep 17 16:02:27.112567 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.112508    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 16:02:27.154205 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.154184    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 16:02:27.227023 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.226987    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume\") pod \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") "
Sep 17 16:02:27.227222 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.227038    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume\") pod \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") "
Sep 17 16:02:27.227222 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.227113    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn4bx\" (UniqueName: \"kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx\") pod \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\" (UID: \"5131578a-4b62-4d1e-aa71-3f61125ed3cb\") "
Sep 17 16:02:27.227222 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.227140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgw7f\" (UniqueName: \"kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f\") pod \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\" (UID: \"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd\") "
Sep 17 16:02:27.227460 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.227430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" (UID: "4ba772c2-ba7d-405d-87d3-a0d9604bd4dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:27.227460 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.227430    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5131578a-4b62-4d1e-aa71-3f61125ed3cb" (UID: "5131578a-4b62-4d1e-aa71-3f61125ed3cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:02:27.228422 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.228313    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5131578a-4b62-4d1e-aa71-3f61125ed3cb-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:27.228422 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.228336    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:27.229656 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.229631    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx" (OuterVolumeSpecName: "kube-api-access-qn4bx") pod "5131578a-4b62-4d1e-aa71-3f61125ed3cb" (UID: "5131578a-4b62-4d1e-aa71-3f61125ed3cb"). InnerVolumeSpecName "kube-api-access-qn4bx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:02:27.229656 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.229643    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f" (OuterVolumeSpecName: "kube-api-access-bgw7f") pod "4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" (UID: "4ba772c2-ba7d-405d-87d3-a0d9604bd4dd"). InnerVolumeSpecName "kube-api-access-bgw7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:02:27.329633 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.329579    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn4bx\" (UniqueName: \"kubernetes.io/projected/5131578a-4b62-4d1e-aa71-3f61125ed3cb-kube-api-access-qn4bx\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:27.329633 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.329607    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgw7f\" (UniqueName: \"kubernetes.io/projected/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd-kube-api-access-bgw7f\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:27.524948 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.524919    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"]
Sep 17 16:02:27.526901 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.526878    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-420219-cvdlj"]
Sep 17 16:02:27.615899 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.615867    2564 generic.go:356] "Generic (PLEG): container finished" podID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" containerID="af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a" exitCode=0
Sep 17 16:02:27.616049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.615911    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"
Sep 17 16:02:27.616049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.615952    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" event={"ID":"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd","Type":"ContainerDied","Data":"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"}
Sep 17 16:02:27.616049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.615998    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28" event={"ID":"4ba772c2-ba7d-405d-87d3-a0d9604bd4dd","Type":"ContainerDied","Data":"3b6882c911679819c55050112b18820ff97048bd5ed31f43463934c8c165c74e"}
Sep 17 16:02:27.616049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.616026    2564 scope.go:117] "RemoveContainer" containerID="af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"
Sep 17 16:02:27.617362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.617338    2564 generic.go:356] "Generic (PLEG): container finished" podID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" containerID="04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7" exitCode=0
Sep 17 16:02:27.617486 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.617389    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"
Sep 17 16:02:27.617486 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.617422    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" event={"ID":"5131578a-4b62-4d1e-aa71-3f61125ed3cb","Type":"ContainerDied","Data":"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"}
Sep 17 16:02:27.617486 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.617472    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4" event={"ID":"5131578a-4b62-4d1e-aa71-3f61125ed3cb","Type":"ContainerDied","Data":"0ce3d14d549cfd0d95f5e7c2da96bd57452dc8c84d38e1a8c5e54acf87138c88"}
Sep 17 16:02:27.626123 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.626102    2564 scope.go:117] "RemoveContainer" containerID="af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"
Sep 17 16:02:27.626397 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:02:27.626373    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a\": container with ID starting with af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a not found: ID does not exist" containerID="af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"
Sep 17 16:02:27.626507 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.626401    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a"} err="failed to get container status \"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a\": rpc error: code = NotFound desc = could not find container \"af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a\": container with ID starting with af07ae86fb7da156153f550422620c43c59c5fb27fef916485e579a242f2f82a not found: ID does not exist"
Sep 17 16:02:27.626507 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.626422    2564 scope.go:117] "RemoveContainer" containerID="04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"
Sep 17 16:02:27.634496 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.634478    2564 scope.go:117] "RemoveContainer" containerID="04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"
Sep 17 16:02:27.634792 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:02:27.634775    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7\": container with ID starting with 04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7 not found: ID does not exist" containerID="04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"
Sep 17 16:02:27.634846 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.634797    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7"} err="failed to get container status \"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7\": rpc error: code = NotFound desc = could not find container \"04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7\": container with ID starting with 04747f07451f22c6d50025d99124c01823ec84419ce409fb10f89df9b1ecc6a7 not found: ID does not exist"
Sep 17 16:02:27.673115 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.673091    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 16:02:27.679455 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.679433    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-420219-68dd79d9fc-29pn4"]
Sep 17 16:02:27.700996 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.700973    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 16:02:27.708179 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:27.708156    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-420219-5798975b54-mgt28"]
Sep 17 16:02:28.526260 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:28.526231    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" path="/var/lib/kubelet/pods/4ba772c2-ba7d-405d-87d3-a0d9604bd4dd/volumes"
Sep 17 16:02:28.526652 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:28.526599    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" path="/var/lib/kubelet/pods/5131578a-4b62-4d1e-aa71-3f61125ed3cb/volumes"
Sep 17 16:02:28.526921 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:28.526906    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0611e31-f0c8-41d3-9bd9-50c351bea08c" path="/var/lib/kubelet/pods/a0611e31-f0c8-41d3-9bd9-50c351bea08c/volumes"
Sep 17 16:02:32.873412 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:32.873379    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:02:37.216955 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:37.216924    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:02:47.892148 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:47.892108    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" podUID="0cf28817-ef1a-4401-a215-387d1b72fd94" containerName="main" containerID="cri-o://f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345" gracePeriod=45
Sep 17 16:02:48.144346 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.144293    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:02:48.332171 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332140    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332191    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332219    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332271    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332300    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwt24\" (UniqueName: \"kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332580 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332395    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs\") pod \"0cf28817-ef1a-4401-a215-387d1b72fd94\" (UID: \"0cf28817-ef1a-4401-a215-387d1b72fd94\") "
Sep 17 16:02:48.332580 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332457    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home" (OuterVolumeSpecName: "home") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.332580 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332466    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache" (OuterVolumeSpecName: "model-cache") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.332580 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332518    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.332797 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332732    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.332797 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332746    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.332797 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.332757    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.334374 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.334340    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm" (OuterVolumeSpecName: "dshm") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:02:48.334496 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.334403    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24" (OuterVolumeSpecName: "kube-api-access-fwt24") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "kube-api-access-fwt24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:02:48.334496 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.334438    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0cf28817-ef1a-4401-a215-387d1b72fd94" (UID: "0cf28817-ef1a-4401-a215-387d1b72fd94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:02:48.433393 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.433317    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0cf28817-ef1a-4401-a215-387d1b72fd94-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.433393 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.433354    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwt24\" (UniqueName: \"kubernetes.io/projected/0cf28817-ef1a-4401-a215-387d1b72fd94-kube-api-access-fwt24\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.433393 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.433369    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cf28817-ef1a-4401-a215-387d1b72fd94-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:02:48.738810 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.738733    2564 generic.go:356] "Generic (PLEG): container finished" podID="0cf28817-ef1a-4401-a215-387d1b72fd94" containerID="f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345" exitCode=0
Sep 17 16:02:48.738810 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.738772    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"
Sep 17 16:02:48.738810 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.738792    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" event={"ID":"0cf28817-ef1a-4401-a215-387d1b72fd94","Type":"ContainerDied","Data":"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"}
Sep 17 16:02:48.739026 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.738822    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs" event={"ID":"0cf28817-ef1a-4401-a215-387d1b72fd94","Type":"ContainerDied","Data":"f88da42268dfc6287cf438284bfe3c0080ebb6b5ffb6fcf6c9f3ff7bc5ef0016"}
Sep 17 16:02:48.739026 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.738839    2564 scope.go:117] "RemoveContainer" containerID="f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"
Sep 17 16:02:48.747506 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.747486    2564 scope.go:117] "RemoveContainer" containerID="f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"
Sep 17 16:02:48.747804 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:02:48.747782    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345\": container with ID starting with f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345 not found: ID does not exist" containerID="f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"
Sep 17 16:02:48.747866 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.747811    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345"} err="failed to get container status \"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345\": rpc error: code = NotFound desc = could not find container \"f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345\": container with ID starting with f207382957d377162779c94418d8cfc9252fd25743f9e8d7e09da92a25bb6345 not found: ID does not exist"
Sep 17 16:02:48.759568 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.759534    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:02:48.765015 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:48.764988    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-420219/ipp-route-420219-kserve-86b64645bc-jjmhs"]
Sep 17 16:02:50.525125 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:02:50.525084    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf28817-ef1a-4401-a215-387d1b72fd94" path="/var/lib/kubelet/pods/0cf28817-ef1a-4401-a215-387d1b72fd94/volumes"
Sep 17 16:03:41.288830 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.288789    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:03:41.289581 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.289070    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" podUID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" containerName="payload-pre-processing" containerID="cri-o://5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd" gracePeriod=30
Sep 17 16:03:41.540634 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.540568    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:03:41.624930 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.624893    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume\") pod \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") "
Sep 17 16:03:41.625199 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.625176    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhtpk\" (UniqueName: \"kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk\") pod \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\" (UID: \"adb5d730-abcd-4d4d-acab-d6fdad40b28a\") "
Sep 17 16:03:41.626136 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.626107    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "adb5d730-abcd-4d4d-acab-d6fdad40b28a" (UID: "adb5d730-abcd-4d4d-acab-d6fdad40b28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:41.628422 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.628393    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk" (OuterVolumeSpecName: "kube-api-access-mhtpk") pod "adb5d730-abcd-4d4d-acab-d6fdad40b28a" (UID: "adb5d730-abcd-4d4d-acab-d6fdad40b28a"). InnerVolumeSpecName "kube-api-access-mhtpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:41.726048 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.726014    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/adb5d730-abcd-4d4d-acab-d6fdad40b28a-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:03:41.726048 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:41.726047    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhtpk\" (UniqueName: \"kubernetes.io/projected/adb5d730-abcd-4d4d-acab-d6fdad40b28a-kube-api-access-mhtpk\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:03:42.017057 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.017018    2564 generic.go:356] "Generic (PLEG): container finished" podID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" containerID="5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd" exitCode=0
Sep 17 16:03:42.017242 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.017061    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"
Sep 17 16:03:42.017242 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.017098    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" event={"ID":"adb5d730-abcd-4d4d-acab-d6fdad40b28a","Type":"ContainerDied","Data":"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"}
Sep 17 16:03:42.017242 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.017152    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj" event={"ID":"adb5d730-abcd-4d4d-acab-d6fdad40b28a","Type":"ContainerDied","Data":"c646297b6b8933fc856f5d8495dec8cbdfe329ed0847cbefe1b782ebea513c04"}
Sep 17 16:03:42.017242 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.017190    2564 scope.go:117] "RemoveContainer" containerID="5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"
Sep 17 16:03:42.027748 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.027721    2564 scope.go:117] "RemoveContainer" containerID="5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"
Sep 17 16:03:42.028047 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:03:42.028024    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd\": container with ID starting with 5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd not found: ID does not exist" containerID="5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"
Sep 17 16:03:42.028128 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.028054    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd"} err="failed to get container status \"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd\": rpc error: code = NotFound desc = could not find container \"5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd\": container with ID starting with 5962301692d15f026cf285bc379e267f2ec001b1680c62c4220951dde2843ccd not found: ID does not exist"
Sep 17 16:03:42.057131 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.057107    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:03:42.063242 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.063219    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddedf5-6c8f8dd5d6-87dpj"]
Sep 17 16:03:42.525644 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:42.525596    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" path="/var/lib/kubelet/pods/adb5d730-abcd-4d4d-acab-d6fdad40b28a/volumes"
Sep 17 16:03:44.658471 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:44.658437    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:03:51.852926 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.852885    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"]
Sep 17 16:03:51.853462 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853442    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" containerName="payload-pre-processing"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853465    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" containerName="payload-pre-processing"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853502    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" containerName="payload-processing"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853511    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" containerName="payload-processing"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853523    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0611e31-f0c8-41d3-9bd9-50c351bea08c" containerName="revoke-keys"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853533    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0611e31-f0c8-41d3-9bd9-50c351bea08c" containerName="revoke-keys"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853545    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" containerName="payload-pre-processing"
Sep 17 16:03:51.853561 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853553    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" containerName="payload-pre-processing"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853569    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf28817-ef1a-4401-a215-387d1b72fd94" containerName="main"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853577    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf28817-ef1a-4401-a215-387d1b72fd94" containerName="main"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853699    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf28817-ef1a-4401-a215-387d1b72fd94" containerName="main"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853713    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5131578a-4b62-4d1e-aa71-3f61125ed3cb" containerName="payload-processing"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853725    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0611e31-f0c8-41d3-9bd9-50c351bea08c" containerName="revoke-keys"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853735    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba772c2-ba7d-405d-87d3-a0d9604bd4dd" containerName="payload-pre-processing"
Sep 17 16:03:51.853937 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.853743    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb5d730-abcd-4d4d-acab-d6fdad40b28a" containerName="payload-pre-processing"
Sep 17 16:03:51.857165 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.857143    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:51.871466 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.871449    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 16:03:51.877771 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.877754    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvk5m\""
Sep 17 16:03:51.895213 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.895191    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"]
Sep 17 16:03:51.925647 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:51.925599    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6\" (UID: \"56254d48-c533-48ce-b621-fb1477bcec7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:52.026453 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:52.026418    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6\" (UID: \"56254d48-c533-48ce-b621-fb1477bcec7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:52.027052 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:52.027032    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6\" (UID: \"56254d48-c533-48ce-b621-fb1477bcec7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:52.166950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:52.166870    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:52.468186 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:52.468163    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"]
Sep 17 16:03:52.470576 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:03:52.470547    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56254d48_c533_48ce_b621_fb1477bcec7c.slice/crio-9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419 WatchSource:0}: Error finding container 9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419: Status 404 returned error can't find the container with id 9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419
Sep 17 16:03:53.079050 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:53.079016    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6" event={"ID":"56254d48-c533-48ce-b621-fb1477bcec7c","Type":"ContainerStarted","Data":"3618e04039a43d880461b94fbe9d0d6d4292d430f81d7ccf70939e71bffbc4e1"}
Sep 17 16:03:53.079429 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:53.079070    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6" event={"ID":"56254d48-c533-48ce-b621-fb1477bcec7c","Type":"ContainerStarted","Data":"9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419"}
Sep 17 16:03:53.120143 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:53.120099    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6" podStartSLOduration=2.120083261 podStartE2EDuration="2.120083261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:03:53.119426006 +0000 UTC m=+2025.086603825" watchObservedRunningTime="2026-09-17 16:03:53.120083261 +0000 UTC m=+2025.087261101"
Sep 17 16:03:54.085946 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:54.085909    2564 generic.go:356] "Generic (PLEG): container finished" podID="56254d48-c533-48ce-b621-fb1477bcec7c" containerID="3618e04039a43d880461b94fbe9d0d6d4292d430f81d7ccf70939e71bffbc4e1" exitCode=0
Sep 17 16:03:54.086309 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:54.086002    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6" event={"ID":"56254d48-c533-48ce-b621-fb1477bcec7c","Type":"ContainerDied","Data":"3618e04039a43d880461b94fbe9d0d6d4292d430f81d7ccf70939e71bffbc4e1"}
Sep 17 16:03:55.229215 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:55.229182    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:55.358680 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:55.358591    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca\") pod \"56254d48-c533-48ce-b621-fb1477bcec7c\" (UID: \"56254d48-c533-48ce-b621-fb1477bcec7c\") "
Sep 17 16:03:55.358958 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:55.358936    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "56254d48-c533-48ce-b621-fb1477bcec7c" (UID: "56254d48-c533-48ce-b621-fb1477bcec7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:55.459977 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:55.459942    2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56254d48-c533-48ce-b621-fb1477bcec7c-maas-api-service-ca\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:03:56.097837 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:56.097809    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"
Sep 17 16:03:56.098033 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:56.097855    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6" event={"ID":"56254d48-c533-48ce-b621-fb1477bcec7c","Type":"ContainerDied","Data":"9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419"}
Sep 17 16:03:56.098033 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:03:56.097900    2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f74436d87503c736303c54fb4cd99f458695f5c6996369be172738fc3b4e419"
Sep 17 16:04:02.664111 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:02.664080    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"]
Sep 17 16:04:02.667814 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:02.667790    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-1d63c0-29827680-g6q27"]
Sep 17 16:04:03.185086 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:03.185052    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"]
Sep 17 16:04:03.188771 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:03.188747    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-1d63c0-rsxz6"]
Sep 17 16:04:04.525907 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:04.525874    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56254d48-c533-48ce-b621-fb1477bcec7c" path="/var/lib/kubelet/pods/56254d48-c533-48ce-b621-fb1477bcec7c/volumes"
Sep 17 16:04:04.526284 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:04.526166    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac8e5d8-5911-4d53-b73d-69ae0cbce654" path="/var/lib/kubelet/pods/fac8e5d8-5911-4d53-b73d-69ae0cbce654/volumes"
Sep 17 16:04:05.810664 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:05.810627    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:04:22.372758 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.372711    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"]
Sep 17 16:04:22.375505 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.375479    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-5818c9-29827680-t6nhp"]
Sep 17 16:04:22.408721 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.408683    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 16:04:22.409090 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.409027    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" podUID="591c1e24-e9d0-4ea8-922c-615548f9db80" containerName="payload-processing" containerID="cri-o://a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88" gracePeriod=30
Sep 17 16:04:22.527490 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.527301    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae690e55-e993-4ce0-b404-45e9f734b2d0" path="/var/lib/kubelet/pods/ae690e55-e993-4ce0-b404-45e9f734b2d0/volumes"
Sep 17 16:04:22.684919 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.684893    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 16:04:22.822898 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.822863    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djhw5\" (UniqueName: \"kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5\") pod \"591c1e24-e9d0-4ea8-922c-615548f9db80\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") "
Sep 17 16:04:22.822898 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.822905    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume\") pod \"591c1e24-e9d0-4ea8-922c-615548f9db80\" (UID: \"591c1e24-e9d0-4ea8-922c-615548f9db80\") "
Sep 17 16:04:22.823343 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.823312    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "591c1e24-e9d0-4ea8-922c-615548f9db80" (UID: "591c1e24-e9d0-4ea8-922c-615548f9db80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:22.825020 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.824994    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5" (OuterVolumeSpecName: "kube-api-access-djhw5") pod "591c1e24-e9d0-4ea8-922c-615548f9db80" (UID: "591c1e24-e9d0-4ea8-922c-615548f9db80"). InnerVolumeSpecName "kube-api-access-djhw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:22.924275 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.924175    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djhw5\" (UniqueName: \"kubernetes.io/projected/591c1e24-e9d0-4ea8-922c-615548f9db80-kube-api-access-djhw5\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:22.924275 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:22.924219    2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/591c1e24-e9d0-4ea8-922c-615548f9db80-plugins-config-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:23.239661 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.239538    2564 generic.go:356] "Generic (PLEG): container finished" podID="591c1e24-e9d0-4ea8-922c-615548f9db80" containerID="a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88" exitCode=0
Sep 17 16:04:23.239661 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.239595    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"
Sep 17 16:04:23.239661 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.239644    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" event={"ID":"591c1e24-e9d0-4ea8-922c-615548f9db80","Type":"ContainerDied","Data":"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"}
Sep 17 16:04:23.239944 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.239697    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh" event={"ID":"591c1e24-e9d0-4ea8-922c-615548f9db80","Type":"ContainerDied","Data":"038bc181e12e3a8cfd861e6cfacb8eec75bcbe91cd01aa9c9ee6882fcc3a754d"}
Sep 17 16:04:23.239944 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.239721    2564 scope.go:117] "RemoveContainer" containerID="a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"
Sep 17 16:04:23.248994 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.248977    2564 scope.go:117] "RemoveContainer" containerID="a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"
Sep 17 16:04:23.249277 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:04:23.249258    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88\": container with ID starting with a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88 not found: ID does not exist" containerID="a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"
Sep 17 16:04:23.249321 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.249285    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88"} err="failed to get container status \"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88\": rpc error: code = NotFound desc = could not find container \"a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88\": container with ID starting with a5bc007da2280f99b6b7e4e434ddb6266e63b6593aea2846b81b94406d909f88 not found: ID does not exist"
Sep 17 16:04:23.275129 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.275092    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 16:04:23.277905 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:23.277881    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-5818c9-658fb58cc-l7bhh"]
Sep 17 16:04:24.081171 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:24.081134    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:04:24.525846 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:24.525813    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="591c1e24-e9d0-4ea8-922c-615548f9db80" path="/var/lib/kubelet/pods/591c1e24-e9d0-4ea8-922c-615548f9db80/volumes"
Sep 17 16:04:30.499059 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:30.499027    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 16:04:30.499450 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:30.499234    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy" containerID="cri-o://2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0" gracePeriod=30
Sep 17 16:04:35.752627 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.752589    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 16:04:35.843932 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.843834    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.843932 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.843884    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.843932 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.843915    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.843945    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.843972    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844019    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844138    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844194    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844232 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844229    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7r65\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65\") pod \"935c4812-9d5e-4d83-b314-aaedd56b9936\" (UID: \"935c4812-9d5e-4d83-b314-aaedd56b9936\") "
Sep 17 16:04:35.844532 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844286    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:35.844532 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844347    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:35.844532 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844363    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:35.844532 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844440    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data" (OuterVolumeSpecName: "istio-data") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:35.844744 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844666    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:04:35.844744 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844710    2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-data\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.844744 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844731    2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.844744 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844744    2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-workload-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.844927 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.844756    2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-credential-socket\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.846225 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.846203    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 16:04:35.846354 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.846333    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65" (OuterVolumeSpecName: "kube-api-access-c7r65") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "kube-api-access-c7r65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:35.846453 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.846431    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token" (OuterVolumeSpecName: "istio-token") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:04:35.846690 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.846671    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "935c4812-9d5e-4d83-b314-aaedd56b9936" (UID: "935c4812-9d5e-4d83-b314-aaedd56b9936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:04:35.945400 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.945359    2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/935c4812-9d5e-4d83-b314-aaedd56b9936-istiod-ca-cert\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.945400 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.945390    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7r65\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-kube-api-access-c7r65\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.945400 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.945400    2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-podinfo\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.945400 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.945409    2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-token\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:35.945746 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:35.945417    2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/935c4812-9d5e-4d83-b314-aaedd56b9936-istio-envoy\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:04:36.128512 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.128422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:04:36.313212 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.313171    2564 generic.go:356] "Generic (PLEG): container finished" podID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerID="2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0" exitCode=0
Sep 17 16:04:36.313412 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.313221    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"
Sep 17 16:04:36.313412 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.313249    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerDied","Data":"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"}
Sep 17 16:04:36.313412 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.313310    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5" event={"ID":"935c4812-9d5e-4d83-b314-aaedd56b9936","Type":"ContainerDied","Data":"9575e57983242626122cd673f9a1cb92c47126430800d16ecfa3f2ee3f3d40dc"}
Sep 17 16:04:36.313412 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.313342    2564 scope.go:117] "RemoveContainer" containerID="2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"
Sep 17 16:04:36.323395 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.323377    2564 scope.go:117] "RemoveContainer" containerID="9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"
Sep 17 16:04:36.332223 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.332200    2564 scope.go:117] "RemoveContainer" containerID="2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"
Sep 17 16:04:36.332535 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:04:36.332513    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0\": container with ID starting with 2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0 not found: ID does not exist" containerID="2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"
Sep 17 16:04:36.332598 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.332543    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0"} err="failed to get container status \"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0\": rpc error: code = NotFound desc = could not find container \"2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0\": container with ID starting with 2d2f78cb308cf19af6b6175b3d99bcba4d7aa35a09f420f78290bf89b32069c0 not found: ID does not exist"
Sep 17 16:04:36.332598 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.332561    2564 scope.go:117] "RemoveContainer" containerID="9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"
Sep 17 16:04:36.332828 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:04:36.332811    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf\": container with ID starting with 9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf not found: ID does not exist" containerID="9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"
Sep 17 16:04:36.332867 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.332832    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf"} err="failed to get container status \"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf\": rpc error: code = NotFound desc = could not find container \"9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf\": container with ID starting with 9feca9933fa43aa78599f1c2029ca611c09d763c6168b76ac6e049199882ccdf not found: ID does not exist"
Sep 17 16:04:36.343465 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.343434    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 16:04:36.346325 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.346297    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-5818c9-openshift-default-57d9b65774-8cpk5"]
Sep 17 16:04:36.525820 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:04:36.525786    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" path="/var/lib/kubelet/pods/935c4812-9d5e-4d83-b314-aaedd56b9936/volumes"
Sep 17 16:05:57.189011 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:05:57.188971    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:06:09.168451 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:06:09.168415    2564 scope.go:117] "RemoveContainer" containerID="c3e3feeafcecec39d1631bec22a92f19223e2027ddf1beea38e6ab9b6415d486"
Sep 17 16:06:09.176904 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:06:09.176883    2564 scope.go:117] "RemoveContainer" containerID="3cdc360b5bba5b2222b7b57dfccaca11100211bd856f53e4febb2f9f1b207b82"
Sep 17 16:06:37.134731 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:06:37.134696    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:06:45.874856 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:06:45.874818    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:06:53.108003 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:06:53.107968    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:07:01.019228 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:01.019194    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:07:20.079050 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:20.079010    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:07:26.074684 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:26.074641    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:07:27.694105 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:27.694068    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 16:07:27.694580 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:27.694287    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" podUID="012d713f-2771-466f-9514-c5aba3196461" containerName="manager" containerID="cri-o://643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34" gracePeriod=10
Sep 17 16:07:28.956487 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:28.956444    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 16:07:28.978609 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:28.978519    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfwsf\" (UniqueName: \"kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf\") pod \"012d713f-2771-466f-9514-c5aba3196461\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") "
Sep 17 16:07:28.978609 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:28.978561    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume\") pod \"012d713f-2771-466f-9514-c5aba3196461\" (UID: \"012d713f-2771-466f-9514-c5aba3196461\") "
Sep 17 16:07:28.978984 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:28.978955    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "012d713f-2771-466f-9514-c5aba3196461" (UID: "012d713f-2771-466f-9514-c5aba3196461"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:28.980807 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:28.980779    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf" (OuterVolumeSpecName: "kube-api-access-sfwsf") pod "012d713f-2771-466f-9514-c5aba3196461" (UID: "012d713f-2771-466f-9514-c5aba3196461"). InnerVolumeSpecName "kube-api-access-sfwsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:29.080148 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.080108    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfwsf\" (UniqueName: \"kubernetes.io/projected/012d713f-2771-466f-9514-c5aba3196461-kube-api-access-sfwsf\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:07:29.080148 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.080140    2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/012d713f-2771-466f-9514-c5aba3196461-extensions-socket-volume\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:07:29.253457 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.253363    2564 generic.go:356] "Generic (PLEG): container finished" podID="012d713f-2771-466f-9514-c5aba3196461" containerID="643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34" exitCode=0
Sep 17 16:07:29.253457 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.253404    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" event={"ID":"012d713f-2771-466f-9514-c5aba3196461","Type":"ContainerDied","Data":"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"}
Sep 17 16:07:29.253457 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.253407    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"
Sep 17 16:07:29.253457 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.253452    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4" event={"ID":"012d713f-2771-466f-9514-c5aba3196461","Type":"ContainerDied","Data":"6aefab9d530f39519432278b6f2844339c3f6966881cc285779e62be83e668ca"}
Sep 17 16:07:29.253768 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.253478    2564 scope.go:117] "RemoveContainer" containerID="643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"
Sep 17 16:07:29.263209 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.263190    2564 scope.go:117] "RemoveContainer" containerID="643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"
Sep 17 16:07:29.263539 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:07:29.263512    2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34\": container with ID starting with 643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34 not found: ID does not exist" containerID="643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"
Sep 17 16:07:29.263657 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.263555    2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34"} err="failed to get container status \"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34\": rpc error: code = NotFound desc = could not find container \"643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34\": container with ID starting with 643d4d88e2881a780bfda4379a369d24a4af5eecf8666e0b31db5578cabfbd34 not found: ID does not exist"
Sep 17 16:07:29.281652 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.281602    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 16:07:29.285549 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:29.285517    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-cvdr4"]
Sep 17 16:07:30.525160 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:07:30.525125    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="012d713f-2771-466f-9514-c5aba3196461" path="/var/lib/kubelet/pods/012d713f-2771-466f-9514-c5aba3196461/volumes"
Sep 17 16:08:09.207386 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:09.207352    2564 scope.go:117] "RemoveContainer" containerID="74dd965608f4cc11e77b86d91cd83e35cff414fbff30595828f7bdb6a8b09e18"
Sep 17 16:08:28.507547 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507512    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"]
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507899    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507910    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507921    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56254d48-c533-48ce-b621-fb1477bcec7c" containerName="revoke-keys"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507927    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="56254d48-c533-48ce-b621-fb1477bcec7c" containerName="revoke-keys"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507941    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="591c1e24-e9d0-4ea8-922c-615548f9db80" containerName="payload-processing"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507946    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="591c1e24-e9d0-4ea8-922c-615548f9db80" containerName="payload-processing"
Sep 17 16:08:28.507950 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507954    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="012d713f-2771-466f-9514-c5aba3196461" containerName="manager"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.507960    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="012d713f-2771-466f-9514-c5aba3196461" containerName="manager"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.508020    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="591c1e24-e9d0-4ea8-922c-615548f9db80" containerName="payload-processing"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.508028    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.508037    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.508043    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="012d713f-2771-466f-9514-c5aba3196461" containerName="manager"
Sep 17 16:08:28.508178 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.508049    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="56254d48-c533-48ce-b621-fb1477bcec7c" containerName="revoke-keys"
Sep 17 16:08:28.510816 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.510799    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.513835 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:08:28.513807    2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" podUID="b2c63c9d-37c7-46f9-8343-722d66ff5d4c" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.515299 ip-10-0-141-17 kubenswrapper[2564]: E0917 16:08:28.515276    2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-j9zv8\" is forbidden: User \"system:node:ip-10-0-141-17.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-17.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j9zv8\"" type="*v1.Secret"
Sep 17 16:08:28.544853 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.544826    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"]
Sep 17 16:08:28.653358 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.653319    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.653550 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.653477    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hgl\" (UniqueName: \"kubernetes.io/projected/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-kube-api-access-66hgl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.754701 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.754669    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.754811 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.754768    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66hgl\" (UniqueName: \"kubernetes.io/projected/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-kube-api-access-66hgl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.755055 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.755037    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:28.767866 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:28.767812    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hgl\" (UniqueName: \"kubernetes.io/projected/b2c63c9d-37c7-46f9-8343-722d66ff5d4c-kube-api-access-66hgl\") pod \"kuadrant-operator-controller-manager-5769f99d9f-5h7xf\" (UID: \"b2c63c9d-37c7-46f9-8343-722d66ff5d4c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:29.821693 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:29.821658    2564 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf" secret="" err="failed to sync secret cache: timed out waiting for the condition"
Sep 17 16:08:29.822076 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:29.821721    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:30.106506 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.106428    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"]
Sep 17 16:08:30.110100 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:08:30.110070    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c63c9d_37c7_46f9_8343_722d66ff5d4c.slice/crio-e7079fce4fb411c2cf9afcacc50b3f21cb52eb833e79fbcf019f0d668314e806 WatchSource:0}: Error finding container e7079fce4fb411c2cf9afcacc50b3f21cb52eb833e79fbcf019f0d668314e806: Status 404 returned error can't find the container with id e7079fce4fb411c2cf9afcacc50b3f21cb52eb833e79fbcf019f0d668314e806
Sep 17 16:08:30.111520 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.111503    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-j9zv8\""
Sep 17 16:08:30.576992 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.576957    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf" event={"ID":"b2c63c9d-37c7-46f9-8343-722d66ff5d4c","Type":"ContainerStarted","Data":"96b0995b016e7ce5f05eac13e3174a1e97c67d53688cbf08e886d7a164eda029"}
Sep 17 16:08:30.577185 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.577006    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf" event={"ID":"b2c63c9d-37c7-46f9-8343-722d66ff5d4c","Type":"ContainerStarted","Data":"e7079fce4fb411c2cf9afcacc50b3f21cb52eb833e79fbcf019f0d668314e806"}
Sep 17 16:08:30.577185 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.577092    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:30.610591 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:30.610538    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf" podStartSLOduration=2.610518433 podStartE2EDuration="2.610518433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:30.603933345 +0000 UTC m=+2302.571111162" watchObservedRunningTime="2026-09-17 16:08:30.610518433 +0000 UTC m=+2302.577696252"
Sep 17 16:08:41.584362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:41.584334    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-5h7xf"
Sep 17 16:08:54.050249 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:08:54.050209    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:02.624756 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:02.624676    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:09.230174 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:09.230144    2564 scope.go:117] "RemoveContainer" containerID="01a8599304219502659abda6903c7055a8f89dfde6709ee069c10e67eea6a12e"
Sep 17 16:09:16.131175 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:16.131131    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:19.971823 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.971787    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:09:19.972211 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.972197    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:09:19.972256 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.972213    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="935c4812-9d5e-4d83-b314-aaedd56b9936" containerName="istio-proxy"
Sep 17 16:09:19.975441 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.975424    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:19.978468 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.978440    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-3c8d1b21-kserve-self-signed-certs\""
Sep 17 16:09:19.988281 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:19.988255    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:09:20.033574 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033542    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmq2\" (UniqueName: \"kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.033765 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033582    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.033765 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033706    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.033765 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033756    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.033928 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033797    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.033928 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.033836    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135105 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135063    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135306 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135198    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135306 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135243    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135306 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135274    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135489 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135313    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135489 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135364    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmq2\" (UniqueName: \"kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135708 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135687    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135816 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135797    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.135909 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.135886    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.137703 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.137681    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.138001 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.137979    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.145022 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.145000    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmq2\" (UniqueName: \"kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2\") pod \"e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") " pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.284944 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.284902    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:20.581787 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.581760    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:09:20.584267 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:09:20.584239    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb76ad7de_c187_458b_b9e1_105acccc70ee.slice/crio-072937d51e441281dba25311a18cbfb128cbd0ac3680da817a09feaf77f47f7d WatchSource:0}: Error finding container 072937d51e441281dba25311a18cbfb128cbd0ac3680da817a09feaf77f47f7d: Status 404 returned error can't find the container with id 072937d51e441281dba25311a18cbfb128cbd0ac3680da817a09feaf77f47f7d
Sep 17 16:09:20.842927 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.842833    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" event={"ID":"b76ad7de-c187-458b-b9e1-105acccc70ee","Type":"ContainerStarted","Data":"eea168012817e3ae663637eebe6467fd1931b6065ddf7fd4eccb8ef6ca3d942a"}
Sep 17 16:09:20.842927 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.842880    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" event={"ID":"b76ad7de-c187-458b-b9e1-105acccc70ee","Type":"ContainerStarted","Data":"072937d51e441281dba25311a18cbfb128cbd0ac3680da817a09feaf77f47f7d"}
Sep 17 16:09:20.890169 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:20.885492    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" podStartSLOduration=1.885472355 podStartE2EDuration="1.885472355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:09:20.883821549 +0000 UTC m=+2352.850999367" watchObservedRunningTime="2026-09-17 16:09:20.885472355 +0000 UTC m=+2352.852650174"
Sep 17 16:09:23.985515 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:23.985477    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:30.285523 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:30.285484    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:30.286048 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:30.285542    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:30.297953 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:30.297922    2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:30.911211 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:30.911178    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:42.842049 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:42.842015    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:09:43.193402 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:43.193304    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:52.387783 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:52.387744    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:09:57.860898 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:57.860857    2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" podUID="b76ad7de-c187-458b-b9e1-105acccc70ee" containerName="main" containerID="cri-o://eea168012817e3ae663637eebe6467fd1931b6065ddf7fd4eccb8ef6ca3d942a" gracePeriod=45
Sep 17 16:09:58.044487 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.044455    2564 generic.go:356] "Generic (PLEG): container finished" podID="b76ad7de-c187-458b-b9e1-105acccc70ee" containerID="eea168012817e3ae663637eebe6467fd1931b6065ddf7fd4eccb8ef6ca3d942a" exitCode=0
Sep 17 16:09:58.044673 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.044501    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" event={"ID":"b76ad7de-c187-458b-b9e1-105acccc70ee","Type":"ContainerDied","Data":"eea168012817e3ae663637eebe6467fd1931b6065ddf7fd4eccb8ef6ca3d942a"}
Sep 17 16:09:58.106313 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.106284    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:58.107469 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107443    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107592 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107487    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szmq2\" (UniqueName: \"kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107592 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107546    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107592 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107578    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107787 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107697    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107787 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107757    2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs\") pod \"b76ad7de-c187-458b-b9e1-105acccc70ee\" (UID: \"b76ad7de-c187-458b-b9e1-105acccc70ee\") "
Sep 17 16:09:58.107940 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107893    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache" (OuterVolumeSpecName: "model-cache") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:58.107940 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.107901    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home" (OuterVolumeSpecName: "home") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:58.108076 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.108021    2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-model-cache\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:58.108076 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.108036    2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-home\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:58.108076 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.108032    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:58.109785 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.109756    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:58.109913 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.109787    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2" (OuterVolumeSpecName: "kube-api-access-szmq2") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "kube-api-access-szmq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:58.110160 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.110145    2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm" (OuterVolumeSpecName: "dshm") pod "b76ad7de-c187-458b-b9e1-105acccc70ee" (UID: "b76ad7de-c187-458b-b9e1-105acccc70ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:58.208669 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.208631    2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-dshm\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:58.208669 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.208670    2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmq2\" (UniqueName: \"kubernetes.io/projected/b76ad7de-c187-458b-b9e1-105acccc70ee-kube-api-access-szmq2\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:58.208889 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.208686    2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b76ad7de-c187-458b-b9e1-105acccc70ee-tmp-dir\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:58.208889 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:58.208700    2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b76ad7de-c187-458b-b9e1-105acccc70ee-tls-certs\") on node \"ip-10-0-141-17.ec2.internal\" DevicePath \"\""
Sep 17 16:09:59.051489 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:59.051461    2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"
Sep 17 16:09:59.051489 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:59.051475    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5" event={"ID":"b76ad7de-c187-458b-b9e1-105acccc70ee","Type":"ContainerDied","Data":"072937d51e441281dba25311a18cbfb128cbd0ac3680da817a09feaf77f47f7d"}
Sep 17 16:09:59.051983 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:59.051521    2564 scope.go:117] "RemoveContainer" containerID="eea168012817e3ae663637eebe6467fd1931b6065ddf7fd4eccb8ef6ca3d942a"
Sep 17 16:09:59.078468 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:59.078441    2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:09:59.081693 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:09:59.081668    2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-3c8d1b21-kserve-5fd9f596ff-hnvn5"]
Sep 17 16:10:00.534245 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:00.534208    2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76ad7de-c187-458b-b9e1-105acccc70ee" path="/var/lib/kubelet/pods/b76ad7de-c187-458b-b9e1-105acccc70ee/volumes"
Sep 17 16:10:02.039646 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:02.039592    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:10:09.249007 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:09.248916    2564 scope.go:117] "RemoveContainer" containerID="3618e04039a43d880461b94fbe9d0d6d4292d430f81d7ccf70939e71bffbc4e1"
Sep 17 16:10:09.903459 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:09.903422    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:10:16.774876 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:16.774837    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:10:22.588182 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:22.588146    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:10:29.886729 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:29.886694    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-266gb"]
Sep 17 16:10:38.295773 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:38.295738    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vqx65_0b3630db-aaf8-4afa-9b5b-1676376ed7f7/manager/0.log"
Sep 17 16:10:38.888388 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:38.888356    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ctnnt_86d1a699-cf68-4442-b8d1-1b1c85c32726/manager/0.log"
Sep 17 16:10:39.296261 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:39.296233    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-ljl7k_77f44441-f9ee-43b6-ace4-f1432cf0b840/copy-manifests/0.log"
Sep 17 16:10:39.305054 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:39.305030    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-ljl7k_77f44441-f9ee-43b6-ace4-f1432cf0b840/manager/0.log"
Sep 17 16:10:39.932362 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:39.932329    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827665-fm2kh_f545ab57-26f0-4b1c-8061-2c4e4a39fde2/cleanup/0.log"
Sep 17 16:10:40.059497 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:40.059457    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827680-jsks9_633ed79b-ee89-45f6-9b68-adda1e74d51f/cleanup/0.log"
Sep 17 16:10:41.504288 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:41.504248    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5h7xf_b2c63c9d-37c7-46f9-8343-722d66ff5d4c/manager/0.log"
Sep 17 16:10:41.648673 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:41.648645    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-266gb_10016316-f547-4d86-a073-1fc468ec9c55/limitador/0.log"
Sep 17 16:10:41.793083 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:41.793055    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dd6ms_3fe3e007-296a-4f1e-8a88-1467d2f0c8cf/manager/0.log"
Sep 17 16:10:42.162125 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:42.162046    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz_4daa9b53-6d59-4199-8d2a-739b7b2306c8/istio-proxy/0.log"
Sep 17 16:10:42.295533 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:42.295501    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gv5kp_af846e86-77ae-4292-aaf4-38824b3dfe77/discovery/0.log"
Sep 17 16:10:42.415036 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:42.414959    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-hvz5n_baad6870-412b-4dc3-a310-87f370489f3e/kube-auth-proxy/0.log"
Sep 17 16:10:42.678564 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:42.678482    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/1.log"
Sep 17 16:10:43.705076 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:43.705044    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-cqscm_81d320fc-7721-4649-bdb8-fa41d453aa6b/main/0.log"
Sep 17 16:10:44.186533 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:44.186503    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-22g7j_46ec7d99-22be-4596-89c2-f9eaa953febb/main/0.log"
Sep 17 16:10:53.651441 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:53.651409    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kg4qt_c2cef1e7-01db-4378-ac73-9efa6ba71ee5/global-pull-secret-syncer/0.log"
Sep 17 16:10:53.860438 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:53.860405    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vbfj6_9b4272a0-bb05-4cb8-879e-afacd2edb750/konnectivity-agent/0.log"
Sep 17 16:10:53.987715 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:53.987632    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-17.ec2.internal_27dcf26562be69b4ba8f5d6ac22207fe/haproxy/0.log"
Sep 17 16:10:58.428438 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:58.428407    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-5h7xf_b2c63c9d-37c7-46f9-8343-722d66ff5d4c/manager/0.log"
Sep 17 16:10:58.536037 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:58.536011    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-266gb_10016316-f547-4d86-a073-1fc468ec9c55/limitador/0.log"
Sep 17 16:10:58.616877 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:10:58.616846    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dd6ms_3fe3e007-296a-4f1e-8a88-1467d2f0c8cf/manager/0.log"
Sep 17 16:11:00.250296 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.250255    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/alertmanager/0.log"
Sep 17 16:11:00.284253 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.284165    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/config-reloader/0.log"
Sep 17 16:11:00.312339 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.312303    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/kube-rbac-proxy-web/0.log"
Sep 17 16:11:00.336590 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.336563    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/kube-rbac-proxy/0.log"
Sep 17 16:11:00.368239 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.368206    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/kube-rbac-proxy-metric/0.log"
Sep 17 16:11:00.396293 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.396264    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/prom-label-proxy/0.log"
Sep 17 16:11:00.450205 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.450162    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b50f3821-df72-4c91-b5f0-0420ee98a03c/init-config-reloader/0.log"
Sep 17 16:11:00.670994 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.670971    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57c57d476b-mccw5_6bb30e77-f930-4311-97a9-602ff4cdb5a4/metrics-server/0.log"
Sep 17 16:11:00.705756 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.705723    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5v27j_7ef6441c-05be-4893-9f4b-97e7a8936a3e/monitoring-plugin/0.log"
Sep 17 16:11:00.946054 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.945978    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9f9p_612681ab-52a7-44bc-8b80-c60399813b50/node-exporter/0.log"
Sep 17 16:11:00.975015 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:00.974989    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9f9p_612681ab-52a7-44bc-8b80-c60399813b50/kube-rbac-proxy/0.log"
Sep 17 16:11:01.010860 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.010836    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p9f9p_612681ab-52a7-44bc-8b80-c60399813b50/init-textfile/0.log"
Sep 17 16:11:01.194909 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.194866    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/prometheus/0.log"
Sep 17 16:11:01.226857 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.226769    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/config-reloader/0.log"
Sep 17 16:11:01.264291 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.264264    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/thanos-sidecar/0.log"
Sep 17 16:11:01.293041 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.293020    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/kube-rbac-proxy-web/0.log"
Sep 17 16:11:01.323011 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.322992    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/kube-rbac-proxy/0.log"
Sep 17 16:11:01.364177 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.364137    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/kube-rbac-proxy-thanos/0.log"
Sep 17 16:11:01.406043 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.406023    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_218ffadf-a8ba-4f8c-bb96-5a9bb0d33da5/init-config-reloader/0.log"
Sep 17 16:11:01.451080 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.451059    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6hft9_9f6537c4-f5a0-47c0-81d6-b13eebc305ab/prometheus-operator/0.log"
Sep 17 16:11:01.477890 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.477802    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-6hft9_9f6537c4-f5a0-47c0-81d6-b13eebc305ab/kube-rbac-proxy/0.log"
Sep 17 16:11:01.639340 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.639309    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/thanos-query/0.log"
Sep 17 16:11:01.668571 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.668548    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/kube-rbac-proxy-web/0.log"
Sep 17 16:11:01.699357 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.699322    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/kube-rbac-proxy/0.log"
Sep 17 16:11:01.726388 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.726367    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/prom-label-proxy/0.log"
Sep 17 16:11:01.762291 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.762269    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/kube-rbac-proxy-rules/0.log"
Sep 17 16:11:01.788272 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:01.788235    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7567c9f887-rqvrf_fdb56917-1441-414c-8e07-87181ad6835e/kube-rbac-proxy-metrics/0.log"
Sep 17 16:11:03.605275 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.605241    2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"]
Sep 17 16:11:03.605756 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.605630    2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76ad7de-c187-458b-b9e1-105acccc70ee" containerName="main"
Sep 17 16:11:03.605756 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.605642    2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76ad7de-c187-458b-b9e1-105acccc70ee" containerName="main"
Sep 17 16:11:03.605756 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.605717    2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76ad7de-c187-458b-b9e1-105acccc70ee" containerName="main"
Sep 17 16:11:03.608840 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.608815    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.611215 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.611190    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"openshift-service-ca.crt\""
Sep 17 16:11:03.611352 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.611190    2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-25lkm\"/\"default-dockercfg-bmf5g\""
Sep 17 16:11:03.611917 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.611901    2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-25lkm\"/\"kube-root-ca.crt\""
Sep 17 16:11:03.628228 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.628199    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"]
Sep 17 16:11:03.736909 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.736866    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-podres\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.737093 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.736929    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-sys\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.737093 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.736977    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-lib-modules\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.737187 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.737096    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-proc\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.737187 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.737156    2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbbnf\" (UniqueName: \"kubernetes.io/projected/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-kube-api-access-hbbnf\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838110 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838073    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbbnf\" (UniqueName: \"kubernetes.io/projected/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-kube-api-access-hbbnf\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838283 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838129    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-podres\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838283 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838160    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-sys\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838283 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838191    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-lib-modules\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838283 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838256    2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-proc\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838458 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838304    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-sys\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838458 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838339    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-proc\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838458 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838341    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-podres\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.838458 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.838404    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-lib-modules\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.847384 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.847362    2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbbnf\" (UniqueName: \"kubernetes.io/projected/6ad0d6b6-0089-4e15-a69c-a6b35f37950a-kube-api-access-hbbnf\") pod \"perf-node-gather-daemonset-89rkq\" (UID: \"6ad0d6b6-0089-4e15-a69c-a6b35f37950a\") " pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:03.919094 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:03.919015    2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:04.196581 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.196553    2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"]
Sep 17 16:11:04.199106 ip-10-0-141-17 kubenswrapper[2564]: W0917 16:11:04.199071    2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ad0d6b6_0089_4e15_a69c_a6b35f37950a.slice/crio-08e159ac14a13e5ba6dda7ca8d3f501b73abcd2cc433a0f4d892a8a276f3328e WatchSource:0}: Error finding container 08e159ac14a13e5ba6dda7ca8d3f501b73abcd2cc433a0f4d892a8a276f3328e: Status 404 returned error can't find the container with id 08e159ac14a13e5ba6dda7ca8d3f501b73abcd2cc433a0f4d892a8a276f3328e
Sep 17 16:11:04.394632 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.394583    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq" event={"ID":"6ad0d6b6-0089-4e15-a69c-a6b35f37950a","Type":"ContainerStarted","Data":"e34d259accc870957021ab458798e255f423507eee6763c8b1ebd893d1a395cc"}
Sep 17 16:11:04.394830 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.394666    2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq" event={"ID":"6ad0d6b6-0089-4e15-a69c-a6b35f37950a","Type":"ContainerStarted","Data":"08e159ac14a13e5ba6dda7ca8d3f501b73abcd2cc433a0f4d892a8a276f3328e"}
Sep 17 16:11:04.394830 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.394691    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:04.417429 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.417376    2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq" podStartSLOduration=1.417357188 podStartE2EDuration="1.417357188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:04.417296594 +0000 UTC m=+2456.384474411" watchObservedRunningTime="2026-09-17 16:11:04.417357188 +0000 UTC m=+2456.384535005"
Sep 17 16:11:04.588741 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:04.588710    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-978784f69-msg4d_e85bdbdc-88fe-4014-bd9e-ced865bb69e7/console/0.log"
Sep 17 16:11:06.748598 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:06.748570    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmxw6_1319ecb9-3468-413d-b218-aedd3a513e45/dns/0.log"
Sep 17 16:11:06.776504 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:06.776477    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xmxw6_1319ecb9-3468-413d-b218-aedd3a513e45/kube-rbac-proxy/0.log"
Sep 17 16:11:06.811780 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:06.811754    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6r2g7_0f588baf-81cd-4efb-a767-bdbd544e38d4/dns-node-resolver/0.log"
Sep 17 16:11:07.537292 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:07.537258    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cjwxc_7daf6886-a7af-4b98-834f-5dee8153323a/node-ca/0.log"
Sep 17 16:11:08.763273 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:08.763243    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz_4daa9b53-6d59-4199-8d2a-739b7b2306c8/istio-proxy/0.log"
Sep 17 16:11:09.173886 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:09.173809    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gv5kp_af846e86-77ae-4292-aaf4-38824b3dfe77/discovery/0.log"
Sep 17 16:11:09.202390 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:09.202362    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-hvz5n_baad6870-412b-4dc3-a310-87f370489f3e/kube-auth-proxy/0.log"
Sep 17 16:11:09.275177 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:09.275142    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/0.log"
Sep 17 16:11:09.278100 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:09.278078    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/1.log"
Sep 17 16:11:10.409690 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:10.409660    2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-25lkm/perf-node-gather-daemonset-89rkq"
Sep 17 16:11:10.714466 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:10.714381    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6bztp_b3bebf42-fc2f-4c92-b2ee-73b10cd40e0a/serve-healthcheck-canary/0.log"
Sep 17 16:11:11.335579 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:11.335553    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lz9k_b32f974a-a7c4-43fa-bda6-dc430abacbad/kube-rbac-proxy/0.log"
Sep 17 16:11:11.360579 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:11.360550    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lz9k_b32f974a-a7c4-43fa-bda6-dc430abacbad/exporter/0.log"
Sep 17 16:11:11.386402 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:11.386377    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6lz9k_b32f974a-a7c4-43fa-bda6-dc430abacbad/extractor/0.log"
Sep 17 16:11:13.936414 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:13.936382    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz_4daa9b53-6d59-4199-8d2a-739b7b2306c8/istio-proxy/0.log"
Sep 17 16:11:14.312768 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.312732    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-vqx65_0b3630db-aaf8-4afa-9b5b-1676376ed7f7/manager/0.log"
Sep 17 16:11:14.347278 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.347225    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gv5kp_af846e86-77ae-4292-aaf4-38824b3dfe77/discovery/0.log"
Sep 17 16:11:14.364024 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.363994    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-hvz5n_baad6870-412b-4dc3-a310-87f370489f3e/kube-auth-proxy/0.log"
Sep 17 16:11:14.415835 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.415805    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/0.log"
Sep 17 16:11:14.417147 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.417128    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/1.log"
Sep 17 16:11:14.757815 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.757781    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ctnnt_86d1a699-cf68-4442-b8d1-1b1c85c32726/manager/0.log"
Sep 17 16:11:14.977709 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:14.977603    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-ljl7k_77f44441-f9ee-43b6-ace4-f1432cf0b840/manager/0.log"
Sep 17 16:11:15.010795 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:15.010770    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5df9996869-ljl7k_77f44441-f9ee-43b6-ace4-f1432cf0b840/copy-manifests/0.log"
Sep 17 16:11:16.016992 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.016964    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-27hrz_4daa9b53-6d59-4199-8d2a-739b7b2306c8/istio-proxy/0.log"
Sep 17 16:11:16.360922 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.360787    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-9p67z_797a030a-0f4b-419e-938e-f29965836e7e/openshift-lws-operator/0.log"
Sep 17 16:11:16.447255 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.447223    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-gv5kp_af846e86-77ae-4292-aaf4-38824b3dfe77/discovery/0.log"
Sep 17 16:11:16.469965 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.469938    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5dc8ddcd98-hvz5n_baad6870-412b-4dc3-a310-87f370489f3e/kube-auth-proxy/0.log"
Sep 17 16:11:16.511473 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.511437    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/0.log"
Sep 17 16:11:16.516549 ip-10-0-141-17 kubenswrapper[2564]: I0917 16:11:16.516526    2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-p4swv_821b7cae-1c87-492c-8404-f3495908a164/istio-proxy/1.log"