Sep 11 18:13:38.744083 ip-10-0-128-148 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 18:13:38.744230 ip-10-0-128-148 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 18:13:38.744269 ip-10-0-128-148 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 18:13:38.744901 ip-10-0-128-148 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 18:13:47.893285 ip-10-0-128-148 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 11 18:13:47.893299 ip-10-0-128-148 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0b80a7ea815949539b1cb4d72d0bd8fa --
Sep 11 18:15:28.527537 ip-10-0-128-148 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 18:15:28.919782 ip-10-0-128-148 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:28.919782 ip-10-0-128-148 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 18:15:28.919782 ip-10-0-128-148 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:28.919782 ip-10-0-128-148 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 18:15:28.919782 ip-10-0-128-148 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 18:15:28.920476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.919797    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924291    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924316    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924320    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924323    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924326    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.924321 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924329    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924332    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924335    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924338    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924340    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924346    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924351    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924354    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924357    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924361    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924363    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924367    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924370    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924373    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924376    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924379    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924382    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924385    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924387    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924390    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.924551 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924392    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924401    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924410    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924414    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924416    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924420    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924423    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924425    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924428    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924430    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924433    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924436    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924438    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924442    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924445    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924448    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924451    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924454    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924458    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.925044 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924462    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924465    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924468    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924471    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924474    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924477    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924480    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924482    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924485    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924487    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924490    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924492    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924495    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924498    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924500    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924503    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924507    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924510    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924513    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924515    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.925619 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924518    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924520    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924523    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924525    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924528    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924531    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924533    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924538    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924540    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924543    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924546    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924549    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924551    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924554    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924557    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924561    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924564    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924566    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924569    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924572    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.926133 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924575    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924577    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924580    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924582    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924585    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924588    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924590    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924592    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924595    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924598    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924600    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924603    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924605    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924608    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.924610    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925347    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925356    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925359    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925362    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925366    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.926639 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925369    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925373    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925377    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925380    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925383    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925386    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925389    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925392    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925394    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925397    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925400    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925402    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925406    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925408    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925411    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925414    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925417    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925419    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925422    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925424    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.927110 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925427    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925430    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925433    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925436    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925438    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925441    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925444    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925447    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925450    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925452    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925455    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925459    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925462    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925465    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925467    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925470    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925472    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925475    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925478    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925480    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.927618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925483    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925485    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925488    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925491    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925494    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925497    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925500    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925502    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925506    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925509    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925512    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925515    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925517    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925520    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925523    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925525    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925528    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925531    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925534    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.928106 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925536    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925540    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925544    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925546    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925550    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925553    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925556    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925559    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925562    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925564    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925567    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925570    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925572    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925574    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925577    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925579    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925582    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925584    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925587    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925590    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.928579 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925593    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925595    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925598    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925600    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925603    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925606    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925608    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925611    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925613    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925616    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925619    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925623    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925626    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925629    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.925632    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926021    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926030    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926033    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926036    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.929069 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926040    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926045    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926049    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926053    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926057    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926060    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926062    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926065    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926068    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926070    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926073    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926075    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926078    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926081    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926083    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926086    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926088    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926091    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926094    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.929546 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926096    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926099    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926102    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926105    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926107    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926110    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926112    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926115    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926117    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926120    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926124    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926127    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926129    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926134    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926138    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926141    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926143    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926146    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926149    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926152    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.930011 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926154    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926157    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926159    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926162    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926164    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926167    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926170    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926173    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926175    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926178    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926181    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926183    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926186    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926188    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926191    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926193    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926196    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926198    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926201    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926203    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.930513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926206    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926208    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926212    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926215    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926218    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926220    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926223    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926230    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926233    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926235    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926238    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926240    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926243    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926245    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926248    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926250    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926253    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926278    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926281    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926291    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.931036 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926294    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926296    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926299    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926302    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926304    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926307    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926310    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926312    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926315    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926318    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926320    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926323    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926326    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926328    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926333    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.926336    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926408    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926451    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926458    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926463    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926467    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 18:15:28.931535 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926471    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926475    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926480    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926483    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926486    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926490    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926493    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926497    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926499    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926503    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926506    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926509    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926512    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926517    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926519    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926522    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926526    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926529    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926533    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926535    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926539    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926542    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926545    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926548    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 18:15:28.932049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926551    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926556    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926559    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926563    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926566    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926569    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926572    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926575    2582 flags.go:64] FLAG: --enable-server="true"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926578    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926586    2582 flags.go:64] FLAG: --event-burst="100"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926589    2582 flags.go:64] FLAG: --event-qps="50"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926592    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926595    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926598    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926602    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926604    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926608    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926610    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926613    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926616    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926620    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926623    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926626    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926629    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926632    2582 flags.go:64] FLAG: --feature-gates=""
Sep 11 18:15:28.932635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926638    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926641    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926645    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926648    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926651    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926654    2582 flags.go:64] FLAG: --help="false"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926657    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-128-148.ec2.internal"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926660    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926664    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926667    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926671    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926674    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926677    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926680    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926683    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926686    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926688    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926692    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926694    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926697    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926700    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926703    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926706    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926709    2582 flags.go:64] FLAG: --lock-file=""
Sep 11 18:15:28.933273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926712    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926715    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926717    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926723    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926731    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926734    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926737    2582 flags.go:64] FLAG: --logging-format="text"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926740    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926744    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926747    2582 flags.go:64] FLAG: --manifest-url=""
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926750    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926755    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926757    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926761    2582 flags.go:64] FLAG: --max-pods="110"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926764    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926767    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926770    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926773    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926777    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926779    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926783    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926790    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926793    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926796    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 18:15:28.933911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926799    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926802    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926808    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926811    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926814    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926816    2582 flags.go:64] FLAG: --port="10250"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926819    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926822    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0751c7e4e684b2e1e"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926826    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926829    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926831    2582 flags.go:64] FLAG: --register-node="true"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926834    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926838    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926841    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926844    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926847    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926851    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926856    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926859    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926862    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926864    2582 flags.go:64] FLAG: --runonce="false"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926867    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926870    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926873    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926876    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 18:15:28.934553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926879    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926882    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926885    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926888    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926891    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926894    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926897    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926900    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926903    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926906    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926909    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926915    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926917    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926920    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926926    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926929    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926932    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926935    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926938    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926940    2582 flags.go:64] FLAG: --v="2"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926945    2582 flags.go:64] FLAG: --version="false"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926956    2582 flags.go:64] FLAG: --vmodule=""
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926963    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.926967    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927095    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.935163 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927098    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927103    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927106    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927109    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927112    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927115    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927118    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927121    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927124    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927127    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927130    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927132    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927135    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927139    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927143    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927146    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927149    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927152    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927154    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.935773 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927158    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927160    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927163    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927166    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927169    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927171    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927174    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927177    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927179    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927182    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927185    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927187    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927192    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927194    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927198    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927202    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927205    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927208    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927211    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.936266 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927215    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927218    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927220    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927223    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927226    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927228    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927231    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927233    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927236    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927238    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927241    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927244    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927246    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927249    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927251    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927254    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927273    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927276    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927278    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927281    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.936745 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927283    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927286    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927290    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927292    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927296    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927300    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927303    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927305    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927308    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927311    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927313    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927316    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927320    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927323    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927325    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927328    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927331    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927333    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927336    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927339    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.937236 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927342    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927344    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927347    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927350    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927352    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927355    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927357    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927360    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927363    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927365    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927368    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927371    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927373    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927376    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927378    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927381    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927384    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927388    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927391    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.937777 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.927394    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.927914    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.935018    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.935040    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.935651    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.935658    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935722    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935727    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935731    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935734    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935737    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935740    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935743    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935745    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935748    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.938298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935751    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935753    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935756    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935758    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935761    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935764    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935767    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935770    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935773    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935777    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935781    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935785    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935788    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935790    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935793    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935796    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935799    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935801    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935804    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.938713 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935807    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935816    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935819    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935822    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935825    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935827    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935830    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935834    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935837    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935840    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935844    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935847    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935850    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935853    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935856    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935858    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935861    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935864    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935866    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.939243 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935869    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935872    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935874    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935878    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935880    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935883    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935885    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935888    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935890    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935893    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935896    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935898    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935901    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935904    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935907    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935916    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935918    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935921    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935924    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935926    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.939794 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935929    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935931    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935934    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935936    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935939    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935942    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935944    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935947    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935949    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935952    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935955    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935957    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935960    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935962    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935965    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935967    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935970    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935972    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935975    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935978    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.940322 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935980    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935983    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935985    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935988    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935990    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935993    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935997    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.935999    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936008    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936011    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936014    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936016    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.936021    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936170    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936175    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 18:15:28.940848 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936178    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936181    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936184    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936187    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936190    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936192    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936195    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936199    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936202    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936204    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936207    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936210    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936212    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936215    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936217    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936220    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936222    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936225    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936228    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936231    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 18:15:28.941228 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936234    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936236    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936239    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936243    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936245    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936248    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936267    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936270    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936273    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936275    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936278    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936281    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936284    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936287    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936290    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936292    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936295    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936298    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936300    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 18:15:28.942031 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936304    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936308    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936311    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936314    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936317    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936319    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936322    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936325    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936327    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936330    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936332    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936335    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936339    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936342    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936345    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936348    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936351    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936354    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936357    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 18:15:28.942832 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936361    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936370    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936373    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936376    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936379    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936382    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936384    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936387    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936390    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936392    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936395    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936397    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936400    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936402    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936405    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936408    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936410    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936413    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936416    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936418    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 18:15:28.943365 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936421    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936423    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936426    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936428    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936431    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936433    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936436    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936438    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936441    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936444    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936446    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936449    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936451    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936454    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936462    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936465    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936468    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936470    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 18:15:28.944064 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:28.936473    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.936477    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.936641    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.938584    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.939595    2582 server.go:1019] "Starting client certificate rotation"
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.939727    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.939789    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 18:15:28.944555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.942210    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:28.946217 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.946187    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 18:15:28.958111 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.958095    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 11 18:15:28.960974 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.960959    2582 log.go:25] "Validated CRI v1 image API"
Sep 11 18:15:28.961945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.961931    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 18:15:28.965786 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.965766    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 90362e07-e0da-4eb4-948a-cdd43f4854ad:/dev/nvme0n1p4 c37f86ab-8127-4ba5-bf5c-baf48bcc6866:/dev/nvme0n1p3]
Sep 11 18:15:28.965826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.965787    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 18:15:28.971105 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.970981    2582 manager.go:216] Machine: {Timestamp:2026-09-11 18:15:28.969781379 +0000 UTC m=+0.328665098 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098686 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21ffa1970e1e7f43ce6fdc6c76b9f6 SystemUUID:ec21ffa1-970e-1e7f-43ce-6fdc6c76b9f6 BootID:0b80a7ea-8159-4953-9b1c-b4d72d0bd8fa Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:85:eb:1f:c2:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:85:eb:1f:c2:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:21:17:f5:fe:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 18:15:28.971105 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.971098    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 18:15:28.971233 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.971181    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 18:15:28.972255 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.972231    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 18:15:28.972421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.972268    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-148.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 18:15:28.973430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.973420    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 18:15:28.973468 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.973432    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 18:15:28.973468 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.973440    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:28.973889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.973873    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:28.974292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.974280    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 18:15:28.975031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.975022    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 18:15:28.978748 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978737    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:28.978862 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978852    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 18:15:28.978894 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978883    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 18:15:28.978924 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978897    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 18:15:28.978963 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978947    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 18:15:28.978992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978963    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 11 18:15:28.978992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.978983    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 18:15:28.979508 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.979496    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:28.979544 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.979517    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 18:15:28.982056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.982022    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 18:15:28.984327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.984313    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 18:15:28.984371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.984343    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 18:15:28.986158 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986144    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986162    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986168    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986174    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986183    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986190    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986203    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986210    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986217    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986222    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986231    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 18:15:28.986238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.986237    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 18:15:28.987691 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.987680    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 18:15:28.987691 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.987690    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 18:15:28.991730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.991708    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-148.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 18:15:28.991911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.991899    2582 server.go:1268] "Started kubelet"
Sep 11 18:15:28.992739 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.992007    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 18:15:28.992798 ip-10-0-128-148 systemd[1]: Started Kubernetes Kubelet.
Sep 11 18:15:28.996349 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.996318    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 18:15:28.996568 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.992352    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 18:15:28.996844 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:28.995959    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 18:15:28.996924 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:28.992701    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 18:15:28.998886 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:28.998857    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 18:15:28.999162 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.999142    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 18:15:28.999369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:28.999350    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 18:15:29.003625 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.003606    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:29.004337 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.003299    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-148.ec2.internal.18d456ed42434426  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-148.ec2.internal,UID:ip-10-0-128-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-148.ec2.internal,},FirstTimestamp:2026-09-11 18:15:28.991872038 +0000 UTC m=+0.350755758,LastTimestamp:2026-09-11 18:15:28.991872038 +0000 UTC m=+0.350755758,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-148.ec2.internal,}"
Sep 11 18:15:29.004337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.004332    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 18:15:29.004536 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.004516    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.004923 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.004906    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 18:15:29.004923 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.004909    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 18:15:29.005056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.004930    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 18:15:29.005056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.005012    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 18:15:29.005056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.005022    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 18:15:29.007335 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.007318    2582 factory.go:55] Registering systemd factory
Sep 11 18:15:29.007433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.007424    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 11 18:15:29.008060 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008045    2582 factory.go:153] Registering CRI-O factory
Sep 11 18:15:29.008134 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008124    2582 factory.go:223] Registration of the crio container factory successfully
Sep 11 18:15:29.008186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008177    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 18:15:29.008235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008205    2582 factory.go:103] Registering Raw factory
Sep 11 18:15:29.008235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008221    2582 manager.go:1196] Started watching for new ooms in manager
Sep 11 18:15:29.008659 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.008644    2582 manager.go:349] Starting recovery of all containers
Sep 11 18:15:29.009705 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.009667    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 18:15:29.009808 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.009723    2582 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-148.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 18:15:29.017421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.017406    2582 manager.go:354] Recovery completed
Sep 11 18:15:29.021457 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.021444    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.029653 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.029635    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.029730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.029670    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.029730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.029683    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.029730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.029717    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sfxs"
Sep 11 18:15:29.030245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.030231    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 18:15:29.030245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.030244    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 18:15:29.030394 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.030236    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.030394 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.030282    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 18:15:29.031497 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.031483    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.031566 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.031513    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.031566 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.031529    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.031820 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.031739    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-148.ec2.internal.18d456ed4483c3a7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-148.ec2.internal,UID:ip-10-0-128-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-148.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-148.ec2.internal,},FirstTimestamp:2026-09-11 18:15:29.029653415 +0000 UTC m=+0.388537133,LastTimestamp:2026-09-11 18:15:29.029653415 +0000 UTC m=+0.388537133,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-148.ec2.internal,}"
Sep 11 18:15:29.032527 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.032512    2582 policy_none.go:49] "None policy: Start"
Sep 11 18:15:29.032599 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.032532    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 18:15:29.032599 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.032545    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 18:15:29.034508 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.034495    2582 policy_none.go:47] "Start"
Sep 11 18:15:29.040415 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.040398    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8sfxs"
Sep 11 18:15:29.044293 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.044210    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-148.ec2.internal.18d456ed44841b3c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-148.ec2.internal,UID:ip-10-0-128-148.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-148.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-148.ec2.internal,},FirstTimestamp:2026-09-11 18:15:29.029675836 +0000 UTC m=+0.388559555,LastTimestamp:2026-09-11 18:15:29.029675836 +0000 UTC m=+0.388559555,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-148.ec2.internal,}"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.072728    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.073925    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.073978    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.074003    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.074046    2582 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.075939    2582 manager.go:337] "Starting Device Plugin manager"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.075980    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.075994    2582 server.go:87] "Starting device plugin registration server"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076425    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076440    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076578    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076647    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076654    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.076835    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.077236    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 18:15:29.087398 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.077298    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.174553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.174466    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"]
Sep 11 18:15:29.174553 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.174553    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.176184 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.176168    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.176238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.176198    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.176238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.176208    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.176536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.176524    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.177364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.177342    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.177455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.177373    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.177455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.177384    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.177455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.177407    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.178426 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.178412    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.178537 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.178525    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.178576 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.178551    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.179179 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179162    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.179179 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179171    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.179321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179194    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.179321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179194    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.179321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179219    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.179321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.179207    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.181578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.181564    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.181635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.181588    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 18:15:29.182333 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.182318    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 18:15:29.182410 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.182346    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 18:15:29.182410 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.182360    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeHasSufficientPID"
Sep 11 18:15:29.186703 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.186689    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.186769 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.186708    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-148.ec2.internal\": node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.202049 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.202025    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-148.ec2.internal\" not found" node="ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.205372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.205355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.205455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.205379    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.205455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.205397    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.206479 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.206459    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-148.ec2.internal\" not found" node="ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.209561 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.209547    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.305893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.305893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305891    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.306102 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305918    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.306102 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.306102 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0a59d530e718667cce688f2a3ad43aaa-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal\" (UID: \"0a59d530e718667cce688f2a3ad43aaa\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.306102 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.305956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f1353260c7ace9243a51756424e4b1aa-config\") pod \"kube-apiserver-proxy-ip-10-0-128-148.ec2.internal\" (UID: \"f1353260c7ace9243a51756424e4b1aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.309961 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.309946    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.410255 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.410220    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.504792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.504720    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.509700 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:29.509431    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:29.510509 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.510493    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.611615 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.611582    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.712168 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.712137    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:29.812645 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:29.812575    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:31.386053 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:31.386021    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-148.ec2.internal\" not found"
Sep 11 18:15:31.412228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.412196    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 18:15:31.412693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.412388    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 18:15:31.457320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.457296    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:31.477528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.477504    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 18:15:31.477528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.477525    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:31.504945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.504920    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:31.507241 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.507221    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 18:15:31.518071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.518044    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 18:10:30 +0000 UTC" deadline="2028-04-24 19:49:38.312035312 +0000 UTC"
Sep 11 18:15:31.518071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.518071    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14185h34m6.793968628s"
Sep 11 18:15:31.542350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.542301    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:31.543869 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.543854    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal"
Sep 11 18:15:31.546899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.546884    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5ljdt"
Sep 11 18:15:31.555374 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.555357    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 18:15:31.556429 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.556415    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5ljdt"
Sep 11 18:15:31.612880 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:31.612810    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1353260c7ace9243a51756424e4b1aa.slice/crio-3eb9ee849aa460433f771bc496f2b8aee88dc6f2e9537befdb04612c04c9e07d WatchSource:0}: Error finding container 3eb9ee849aa460433f771bc496f2b8aee88dc6f2e9537befdb04612c04c9e07d: Status 404 returned error can't find the container with id 3eb9ee849aa460433f771bc496f2b8aee88dc6f2e9537befdb04612c04c9e07d
Sep 11 18:15:31.614732 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:31.614709    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a59d530e718667cce688f2a3ad43aaa.slice/crio-a66c094c143076b91a552727832f643cfab4ca0995a65d87d688b23740f909e4 WatchSource:0}: Error finding container a66c094c143076b91a552727832f643cfab4ca0995a65d87d688b23740f909e4: Status 404 returned error can't find the container with id a66c094c143076b91a552727832f643cfab4ca0995a65d87d688b23740f909e4
Sep 11 18:15:31.617589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.617570    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:15:31.735476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:31.735447    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:32.258697 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.258664    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 18:15:32.452711 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.452682    2582 apiserver.go:52] "Watching apiserver"
Sep 11 18:15:32.464354 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.464321    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 18:15:32.467083 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.467046    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-s7h6h","openshift-image-registry/node-ca-n6pjd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal","openshift-multus/multus-v8bhv","openshift-ovn-kubernetes/ovnkube-node-967q2","kube-system/global-pull-secret-syncer-kd2j8","kube-system/konnectivity-agent-qdmqh","kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln","openshift-multus/multus-additional-cni-plugins-jlbkg","openshift-multus/network-metrics-daemon-2x9w6","openshift-network-diagnostics/network-check-target-7p4mx","openshift-network-operator/iptables-alerter-44bnm","openshift-cluster-node-tuning-operator/tuned-nlqt7"]
Sep 11 18:15:32.469623 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.469603    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.471671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.471645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.472517 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.472494    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 18:15:32.472728 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.472704    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 18:15:32.472813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.472708    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-49gw6\""
Sep 11 18:15:32.474131 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.474112    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.474985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.474958    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.475429 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.475408    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.475546 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.475437    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 18:15:32.475546 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.475532    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hqng8\""
Sep 11 18:15:32.476163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.476148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.477187 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.477053    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.477187 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.477062    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.477343 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.477227    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lkmwq\""
Sep 11 18:15:32.478684 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.478549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.479287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.479267    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.479372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.479315    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 18:15:32.479372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.479351    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.479463 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.479310    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 18:15:32.480246 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.480081    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9m9gh\""
Sep 11 18:15:32.480546 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.480503    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.480798 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.480779    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.480861 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.480823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.480935 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.480897    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:32.481031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.481015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 18:15:32.481701 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.481318    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n7lhd\""
Sep 11 18:15:32.481701 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.481364    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 18:15:32.481701 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.481520    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 18:15:32.481925 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.481795    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 18:15:32.483545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.483528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.485829 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.485809    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:32.485920 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.485871    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:32.486386 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.486364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zprh5\""
Sep 11 18:15:32.486618 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.486608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.486838 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.486819    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 18:15:32.487140 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-conf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487141    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.487261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487153    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-hostroot\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.487261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487200    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-config\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487223    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-host\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487257    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-tuned\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487281    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-socket-dir-parent\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487304    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-slash\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-node-log\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-bin\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-tmp\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-var-lib-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487491    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-bin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487515    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8b33b1-a528-4085-9efe-a4873d6278a8-iptables-alerter-script\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487538    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-systemd\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487589    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-sys\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-ovn\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-modprobe-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487715    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-var-lib-kubelet\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487751    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-systemd-units\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487802    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-run\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.487880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487841    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-netd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpnql\" (UniqueName: \"kubernetes.io/projected/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-kube-api-access-zpnql\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8b33b1-a528-4085-9efe-a4873d6278a8-host-slash\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-conf-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.487989    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-multus-certs\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k95tx\" (UniqueName: \"kubernetes.io/projected/060637b0-6735-4ce5-b082-6c943d98b6d2-kube-api-access-k95tx\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488043    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-netns\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aeadb79-70da-4d8e-a32c-276592debddb-konnectivity-ca\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488092    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-env-overrides\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488115    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aeadb79-70da-4d8e-a32c-276592debddb-agent-certs\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-lib-modules\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488204    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-etc-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488229    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488269    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovn-node-metrics-cert\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488307    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67cp\" (UniqueName: \"kubernetes.io/projected/cc8b33b1-a528-4085-9efe-a4873d6278a8-kube-api-access-k67cp\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.488352 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-daemon-config\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488352    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-log-socket\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488375    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-cni-binary-copy\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488406    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-system-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-kubelet\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-kubernetes\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-cnibin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488671    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-netns\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-kubelet\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488741    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8kcf\" (UniqueName: \"kubernetes.io/projected/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-kube-api-access-z8kcf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488763    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-k8s-cni-cncf-io\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-multus\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488809    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-etc-kubernetes\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-script-lib\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-systemd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.489019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.488900    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysconfig\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.490143 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.490124    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.490315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.490300    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 18:15:32.490448 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.490429    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x45gx\""
Sep 11 18:15:32.490594 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.490575    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.490743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.490704    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.493078 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.493050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.493198 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.493122    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:32.494918 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.494895    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zddnf\""
Sep 11 18:15:32.495003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.494987    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 18:15:32.495117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.495089    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 18:15:32.496705 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.496680    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.498598 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.498582    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 18:15:32.498780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.498761    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qx7q7\""
Sep 11 18:15:32.498906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.498890    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 18:15:32.550992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.550945    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" event={"ID":"f1353260c7ace9243a51756424e4b1aa","Type":"ContainerStarted","Data":"3eb9ee849aa460433f771bc496f2b8aee88dc6f2e9537befdb04612c04c9e07d"}
Sep 11 18:15:32.552158 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.552136    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerStarted","Data":"a66c094c143076b91a552727832f643cfab4ca0995a65d87d688b23740f909e4"}
Sep 11 18:15:32.559350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.559296    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:31 +0000 UTC" deadline="2028-06-14 00:29:39.13840979 +0000 UTC"
Sep 11 18:15:32.559350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.559344    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15390h14m6.579069209s"
Sep 11 18:15:32.579754 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.579539    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 18:15:32.589332 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589275    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cnibin\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.589430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589353    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysconfig\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-conf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-hostroot\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.589430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-config\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-host\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-tuned\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-hostroot\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-socket-dir-parent\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589547    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-host\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589561    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-conf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-slash\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysconfig\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-socket-dir-parent\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589647    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-slash\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-node-log\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.589722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589700    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-bin\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-node-log\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589755    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589773    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-bin\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589785    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-tmp\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-var-lib-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589856    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44dd8471-2506-4cc9-9c44-9f9572550751-tmp-dir\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589881    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-dbus\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-var-lib-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589915    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589945    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-bin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8b33b1-a528-4085-9efe-a4873d6278a8-iptables-alerter-script\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.589994    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-systemd\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590000    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-sys\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590042    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-ovn\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590077    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-systemd\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590093    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-config\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590015    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-bin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-etc-selinux\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590139    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590145    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-ovn\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590181    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590199    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-sys\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdv9b\" (UniqueName: \"kubernetes.io/projected/44dd8471-2506-4cc9-9c44-9f9572550751-kube-api-access-zdv9b\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590208    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590233    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-modprobe-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590245    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-var-lib-kubelet\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590304    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-systemd-units\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590326    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-modprobe-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp4b5\" (UniqueName: \"kubernetes.io/projected/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kube-api-access-lp4b5\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-systemd-units\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a948bb34-43b3-4f19-8cd0-0851b7f71882-host\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.590956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-var-lib-kubelet\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590426    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-run\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590451    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-netd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-cni-netd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-run\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590537    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc8b33b1-a528-4085-9efe-a4873d6278a8-iptables-alerter-script\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590558    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpnql\" (UniqueName: \"kubernetes.io/projected/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-kube-api-access-zpnql\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590621    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-kubelet-config\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8b33b1-a528-4085-9efe-a4873d6278a8-host-slash\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590783    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc8b33b1-a528-4085-9efe-a4873d6278a8-host-slash\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-conf-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-multus-certs\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590906    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k95tx\" (UniqueName: \"kubernetes.io/projected/060637b0-6735-4ce5-b082-6c943d98b6d2-kube-api-access-k95tx\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590926    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-netns\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aeadb79-70da-4d8e-a32c-276592debddb-konnectivity-ca\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.591802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590973    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-multus-certs\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591000    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-env-overrides\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591018    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-conf-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aeadb79-70da-4d8e-a32c-276592debddb-agent-certs\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-lib-modules\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-etc-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591130    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovn-node-metrics-cert\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591195    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-socket-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591226    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds7lq\" (UniqueName: \"kubernetes.io/projected/a948bb34-43b3-4f19-8cd0-0851b7f71882-kube-api-access-ds7lq\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591247    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xprhw\" (UniqueName: \"kubernetes.io/projected/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-kube-api-access-xprhw\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591270    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67cp\" (UniqueName: \"kubernetes.io/projected/cc8b33b1-a528-4085-9efe-a4873d6278a8-kube-api-access-k67cp\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-daemon-config\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-netns\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.592622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591337    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-log-socket\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591372    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-sys-fs\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591393    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-cni-binary-copy\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-system-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-kubelet\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-device-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591520    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-system-cni-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-kubernetes\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591565    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-cnibin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591590    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-netns\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-kubelet\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591678    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a948bb34-43b3-4f19-8cd0-0851b7f71882-serviceca\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2aeadb79-70da-4d8e-a32c-276592debddb-konnectivity-ca\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.590946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-sysctl-d\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88r84\" (UniqueName: \"kubernetes.io/projected/972c3842-6619-4463-802a-7a7ca6e6db54-kube-api-access-88r84\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.593476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8kcf\" (UniqueName: \"kubernetes.io/projected/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-kube-api-access-z8kcf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-k8s-cni-cncf-io\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-multus\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-etc-kubernetes\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591843    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-system-cni-dir\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-script-lib\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591877    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-registration-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-kubelet\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44dd8471-2506-4cc9-9c44-9f9572550751-hosts-file\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-systemd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-kubernetes\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-systemd\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.591997    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-cnibin\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-netns\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-kubelet\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592116    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-host-run-ovn-kubernetes\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-var-lib-cni-multus\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592251    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-cni-binary-copy\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.594316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592367    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-env-overrides\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-etc-kubernetes\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592907    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovnkube-script-lib\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-log-socket\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.592960    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/060637b0-6735-4ce5-b082-6c943d98b6d2-multus-daemon-config\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.593020    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-lib-modules\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.593033    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/060637b0-6735-4ce5-b082-6c943d98b6d2-host-run-k8s-cni-cncf-io\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.593035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-etc-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.593072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-run-openvswitch\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.594241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-etc-tuned\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.594345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-tmp\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.595155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.594932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2aeadb79-70da-4d8e-a32c-276592debddb-agent-certs\") pod \"konnectivity-agent-qdmqh\" (UID: \"2aeadb79-70da-4d8e-a32c-276592debddb\") " pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.595552 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.595241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-ovn-node-metrics-cert\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.599224 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.599194    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k95tx\" (UniqueName: \"kubernetes.io/projected/060637b0-6735-4ce5-b082-6c943d98b6d2-kube-api-access-k95tx\") pod \"multus-v8bhv\" (UID: \"060637b0-6735-4ce5-b082-6c943d98b6d2\") " pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.599737 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.599717    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpnql\" (UniqueName: \"kubernetes.io/projected/a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d-kube-api-access-zpnql\") pod \"ovnkube-node-967q2\" (UID: \"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d\") " pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.601140 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.600996    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67cp\" (UniqueName: \"kubernetes.io/projected/cc8b33b1-a528-4085-9efe-a4873d6278a8-kube-api-access-k67cp\") pod \"iptables-alerter-44bnm\" (UID: \"cc8b33b1-a528-4085-9efe-a4873d6278a8\") " pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.601977 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.601955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8kcf\" (UniqueName: \"kubernetes.io/projected/fecc12ff-46ac-4ca5-8a23-c6b426446bb2-kube-api-access-z8kcf\") pod \"tuned-nlqt7\" (UID: \"fecc12ff-46ac-4ca5-8a23-c6b426446bb2\") " pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.692581 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44dd8471-2506-4cc9-9c44-9f9572550751-tmp-dir\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.692581 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692580    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-dbus\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692608    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-etc-selinux\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692636    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692660    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdv9b\" (UniqueName: \"kubernetes.io/projected/44dd8471-2506-4cc9-9c44-9f9572550751-kube-api-access-zdv9b\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692687    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp4b5\" (UniqueName: \"kubernetes.io/projected/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kube-api-access-lp4b5\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-dbus\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a948bb34-43b3-4f19-8cd0-0851b7f71882-host\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692764    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a948bb34-43b3-4f19-8cd0-0851b7f71882-host\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.692806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692787    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-etc-selinux\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692793    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-kubelet-config\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44dd8471-2506-4cc9-9c44-9f9572550751-tmp-dir\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692946    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-kubelet-config\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-socket-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.692989    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.693061    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693089    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds7lq\" (UniqueName: \"kubernetes.io/projected/a948bb34-43b3-4f19-8cd0-0851b7f71882-kube-api-access-ds7lq\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693094    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-socket-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xprhw\" (UniqueName: \"kubernetes.io/projected/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-kube-api-access-xprhw\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.693143    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.193122543 +0000 UTC m=+3.079095035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:32.693220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-sys-fs\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693222    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-device-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693276    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-sys-fs\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693291    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-binary-copy\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.693319    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693301    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-device-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-system-cni-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.693376    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.193359106 +0000 UTC m=+3.079331587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693396    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-system-cni-dir\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693416    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a948bb34-43b3-4f19-8cd0-0851b7f71882-serviceca\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693435    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88r84\" (UniqueName: \"kubernetes.io/projected/972c3842-6619-4463-802a-7a7ca6e6db54-kube-api-access-88r84\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693456    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-registration-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693477    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44dd8471-2506-4cc9-9c44-9f9572550751-hosts-file\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cnibin\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693556    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44dd8471-2506-4cc9-9c44-9f9572550751-hosts-file\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693579    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cnibin\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f22573dd-73e1-47ad-8bd4-0aac29bcb589-registration-dir\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.693797    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a948bb34-43b3-4f19-8cd0-0851b7f71882-serviceca\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.694673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.694046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.704438 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.704417    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:32.704438 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.704439    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:32.704616 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.704453    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:32.704616 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:32.704527    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:33.20451107 +0000 UTC m=+3.090483542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:32.705345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.705320    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp4b5\" (UniqueName: \"kubernetes.io/projected/f22573dd-73e1-47ad-8bd4-0aac29bcb589-kube-api-access-lp4b5\") pod \"aws-ebs-csi-driver-node-j9cln\" (UID: \"f22573dd-73e1-47ad-8bd4-0aac29bcb589\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.706571 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.706498    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88r84\" (UniqueName: \"kubernetes.io/projected/972c3842-6619-4463-802a-7a7ca6e6db54-kube-api-access-88r84\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:32.706772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.706754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds7lq\" (UniqueName: \"kubernetes.io/projected/a948bb34-43b3-4f19-8cd0-0851b7f71882-kube-api-access-ds7lq\") pod \"node-ca-n6pjd\" (UID: \"a948bb34-43b3-4f19-8cd0-0851b7f71882\") " pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.706867 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.706848    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xprhw\" (UniqueName: \"kubernetes.io/projected/fa4965d6-e8d5-4227-a78f-d28d3bc2278d-kube-api-access-xprhw\") pod \"multus-additional-cni-plugins-jlbkg\" (UID: \"fa4965d6-e8d5-4227-a78f-d28d3bc2278d\") " pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.706982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.706959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdv9b\" (UniqueName: \"kubernetes.io/projected/44dd8471-2506-4cc9-9c44-9f9572550751-kube-api-access-zdv9b\") pod \"node-resolver-s7h6h\" (UID: \"44dd8471-2506-4cc9-9c44-9f9572550751\") " pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:32.782688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.782567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:32.792112 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.792080    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-44bnm"
Sep 11 18:15:32.800070 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.800045    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlqt7"
Sep 11 18:15:32.804928 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.804906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-v8bhv"
Sep 11 18:15:32.810637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.810619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:32.817232 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.817214    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln"
Sep 11 18:15:32.822379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.822362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-n6pjd"
Sep 11 18:15:32.828885 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.828862    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jlbkg"
Sep 11 18:15:32.833444 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:32.833428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s7h6h"
Sep 11 18:15:33.197367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.197279    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:33.197367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.197340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:33.197574 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.197444    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:33.197574 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.197474    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:33.197574 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.197514    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.197495791 +0000 UTC m=+4.083468273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:33.197574 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.197537    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.197527044 +0000 UTC m=+4.083499515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:33.298392 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.298357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:33.298554 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.298489    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:33.298554 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.298507    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:33.298554 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.298519    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:33.298662 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.298579    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:34.298566122 +0000 UTC m=+4.184538590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:33.483375 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.483254    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6eb9f67_93e1_45cb_b5c7_c6ebff4dc73d.slice/crio-442c7492b6df88737638fddea0ca85985eef11fd3e87ab339cadf986b0eb1db6 WatchSource:0}: Error finding container 442c7492b6df88737638fddea0ca85985eef11fd3e87ab339cadf986b0eb1db6: Status 404 returned error can't find the container with id 442c7492b6df88737638fddea0ca85985eef11fd3e87ab339cadf986b0eb1db6
Sep 11 18:15:33.485762 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.485735    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4965d6_e8d5_4227_a78f_d28d3bc2278d.slice/crio-53a7514ed41f031839a820e833990634d3a8e58052f61b4e88d6f542bd344625 WatchSource:0}: Error finding container 53a7514ed41f031839a820e833990634d3a8e58052f61b4e88d6f542bd344625: Status 404 returned error can't find the container with id 53a7514ed41f031839a820e833990634d3a8e58052f61b4e88d6f542bd344625
Sep 11 18:15:33.488587 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.488003    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060637b0_6735_4ce5_b082_6c943d98b6d2.slice/crio-a3013551aa8a85edc0c1065737f9368d8c7aaf1986b2c39c38204fd0993679ae WatchSource:0}: Error finding container a3013551aa8a85edc0c1065737f9368d8c7aaf1986b2c39c38204fd0993679ae: Status 404 returned error can't find the container with id a3013551aa8a85edc0c1065737f9368d8c7aaf1986b2c39c38204fd0993679ae
Sep 11 18:15:33.491450 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.491428    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aeadb79_70da_4d8e_a32c_276592debddb.slice/crio-d10145c659bb9643dfbac0e7975d9f3b2d5af282da1842f1666b0047fd2cbffc WatchSource:0}: Error finding container d10145c659bb9643dfbac0e7975d9f3b2d5af282da1842f1666b0047fd2cbffc: Status 404 returned error can't find the container with id d10145c659bb9643dfbac0e7975d9f3b2d5af282da1842f1666b0047fd2cbffc
Sep 11 18:15:33.512109 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.512080    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfecc12ff_46ac_4ca5_8a23_c6b426446bb2.slice/crio-20e4c58bfc38f54c21780fbb29b0e55d19826238d4a3f38e83f67ad03026a7b6 WatchSource:0}: Error finding container 20e4c58bfc38f54c21780fbb29b0e55d19826238d4a3f38e83f67ad03026a7b6: Status 404 returned error can't find the container with id 20e4c58bfc38f54c21780fbb29b0e55d19826238d4a3f38e83f67ad03026a7b6
Sep 11 18:15:33.513415 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.513340    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda948bb34_43b3_4f19_8cd0_0851b7f71882.slice/crio-f200a19131d87d45c20fda65ab8e181f9ffe12ba256ba3737dfaec10fb2a8120 WatchSource:0}: Error finding container f200a19131d87d45c20fda65ab8e181f9ffe12ba256ba3737dfaec10fb2a8120: Status 404 returned error can't find the container with id f200a19131d87d45c20fda65ab8e181f9ffe12ba256ba3737dfaec10fb2a8120
Sep 11 18:15:33.514115 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.514083    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dd8471_2506_4cc9_9c44_9f9572550751.slice/crio-d68f8f5a06aadb2cc08edb742d7d52da55900366344519ba717004335bbf5fcb WatchSource:0}: Error finding container d68f8f5a06aadb2cc08edb742d7d52da55900366344519ba717004335bbf5fcb: Status 404 returned error can't find the container with id d68f8f5a06aadb2cc08edb742d7d52da55900366344519ba717004335bbf5fcb
Sep 11 18:15:33.515153 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.515108    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8b33b1_a528_4085_9efe_a4873d6278a8.slice/crio-a7a87f34f636ac896d321ed6b13a23bf91596379582d101171fd1b1f9a9bd423 WatchSource:0}: Error finding container a7a87f34f636ac896d321ed6b13a23bf91596379582d101171fd1b1f9a9bd423: Status 404 returned error can't find the container with id a7a87f34f636ac896d321ed6b13a23bf91596379582d101171fd1b1f9a9bd423
Sep 11 18:15:33.516022 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:15:33.515824    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22573dd_73e1_47ad_8bd4_0aac29bcb589.slice/crio-a79d801ee846e1342ccd6264f10f3cbb939428d60a7a6891d2131d3c72ca93b8 WatchSource:0}: Error finding container a79d801ee846e1342ccd6264f10f3cbb939428d60a7a6891d2131d3c72ca93b8: Status 404 returned error can't find the container with id a79d801ee846e1342ccd6264f10f3cbb939428d60a7a6891d2131d3c72ca93b8
Sep 11 18:15:33.548088 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.548063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:33.548233 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:33.548206    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:33.556514 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.556489    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln" event={"ID":"f22573dd-73e1-47ad-8bd4-0aac29bcb589","Type":"ContainerStarted","Data":"a79d801ee846e1342ccd6264f10f3cbb939428d60a7a6891d2131d3c72ca93b8"}
Sep 11 18:15:33.558713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.558685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44bnm" event={"ID":"cc8b33b1-a528-4085-9efe-a4873d6278a8","Type":"ContainerStarted","Data":"a7a87f34f636ac896d321ed6b13a23bf91596379582d101171fd1b1f9a9bd423"}
Sep 11 18:15:33.559524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.559473    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 18:10:31 +0000 UTC" deadline="2028-03-14 04:10:53.236251573 +0000 UTC"
Sep 11 18:15:33.559591 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.559524    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13185h55m19.676730767s"
Sep 11 18:15:33.561295 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.561268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7h6h" event={"ID":"44dd8471-2506-4cc9-9c44-9f9572550751","Type":"ContainerStarted","Data":"d68f8f5a06aadb2cc08edb742d7d52da55900366344519ba717004335bbf5fcb"}
Sep 11 18:15:33.562383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.562342    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6pjd" event={"ID":"a948bb34-43b3-4f19-8cd0-0851b7f71882","Type":"ContainerStarted","Data":"f200a19131d87d45c20fda65ab8e181f9ffe12ba256ba3737dfaec10fb2a8120"}
Sep 11 18:15:33.563536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.563502    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlqt7" event={"ID":"fecc12ff-46ac-4ca5-8a23-c6b426446bb2","Type":"ContainerStarted","Data":"20e4c58bfc38f54c21780fbb29b0e55d19826238d4a3f38e83f67ad03026a7b6"}
Sep 11 18:15:33.564847 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.564828    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qdmqh" event={"ID":"2aeadb79-70da-4d8e-a32c-276592debddb","Type":"ContainerStarted","Data":"d10145c659bb9643dfbac0e7975d9f3b2d5af282da1842f1666b0047fd2cbffc"}
Sep 11 18:15:33.566193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.566157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"442c7492b6df88737638fddea0ca85985eef11fd3e87ab339cadf986b0eb1db6"}
Sep 11 18:15:33.567375 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.567351    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8bhv" event={"ID":"060637b0-6735-4ce5-b082-6c943d98b6d2","Type":"ContainerStarted","Data":"a3013551aa8a85edc0c1065737f9368d8c7aaf1986b2c39c38204fd0993679ae"}
Sep 11 18:15:33.568617 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:33.568595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerStarted","Data":"53a7514ed41f031839a820e833990634d3a8e58052f61b4e88d6f542bd344625"}
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.203551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.203613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.203781    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.203848    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.203829476 +0000 UTC m=+6.089801947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.204303    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:34.204486 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.204379    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.204362693 +0000 UTC m=+6.090335161 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:34.304616 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.304576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:34.304850 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.304747    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:34.304850 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.304767    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:34.304850 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.304779    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:34.304850 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.304838    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:36.304819126 +0000 UTC m=+6.190791596 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:34.548918 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.548843    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:34.549854 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.548976    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:34.549854 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.549405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:34.549854 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:34.549520    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:34.593872 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.592813    2582 generic.go:356] "Generic (PLEG): container finished" podID="0a59d530e718667cce688f2a3ad43aaa" containerID="d2b4544aa74ef9031dd6cbf1c04c678229d2a950d4a87b8e6f8398f9fee742d1" exitCode=0
Sep 11 18:15:34.593872 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.592913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerDied","Data":"d2b4544aa74ef9031dd6cbf1c04c678229d2a950d4a87b8e6f8398f9fee742d1"}
Sep 11 18:15:34.611097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:34.610671    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" event={"ID":"f1353260c7ace9243a51756424e4b1aa","Type":"ContainerStarted","Data":"4dc0aad3296a65fb03943c4c16c5fa894ab2c6a1a4e0a640ea66a21d39d1b8ff"}
Sep 11 18:15:35.548197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:35.547693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:35.548197 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:35.547828    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:35.629103 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:35.629031    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" event={"ID":"0a59d530e718667cce688f2a3ad43aaa","Type":"ContainerStarted","Data":"59746267df6de73c3b442a7ca5cb4e439b71f2fa94e9703664a3d67cc12cf016"}
Sep 11 18:15:35.644058 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:35.643398    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-148.ec2.internal" podStartSLOduration=4.64337657 podStartE2EDuration="4.64337657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:34.624262289 +0000 UTC m=+4.510234778" watchObservedRunningTime="2026-09-11 18:15:35.64337657 +0000 UTC m=+5.529349061"
Sep 11 18:15:36.220701 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:36.220436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:36.221005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:36.220784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:36.221005 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.220918    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:36.221005 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.220984    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:40.220966403 +0000 UTC m=+10.106938894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:36.221514 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.221392    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:36.221514 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.221448    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:40.22143136 +0000 UTC m=+10.107403842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:36.322706 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:36.322085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:36.322706 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.322253    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:36.322706 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.322274    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:36.322706 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.322288    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:36.322706 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.322350    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:40.322331951 +0000 UTC m=+10.208304441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:36.548883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:36.548805    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:36.548883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:36.548850    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:36.549081 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.548950    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:36.549136 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:36.549074    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:37.548032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:37.547998    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:37.548431 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:37.548137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:38.548106 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:38.548070    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:38.548551 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:38.548225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:38.548551 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:38.548086    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:38.548551 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:38.548446    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:39.547680 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:39.547596    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:39.547830 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:39.547732    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:40.258625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:40.258696    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.258790    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.258839    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:48.258825644 +0000 UTC m=+18.144798110 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.259159    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:40.259272 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.259229    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:48.259212586 +0000 UTC m=+18.145185068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:40.359579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:40.359486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:40.359771 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.359652    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:40.359771 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.359678    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:40.359771 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.359692    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:40.359771 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.359755    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:15:48.359737117 +0000 UTC m=+18.245709584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:40.549624 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:40.549006    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:40.549624 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:40.549119    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:40.549624 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.549222    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:40.549624 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:40.549246    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:41.547136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:41.547101    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:41.547593 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:41.547263    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:42.547343 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:42.547310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:42.547731 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:42.547443    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:42.547731 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:42.547481    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:42.547731 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:42.547578    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:43.547838 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:43.547801    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:43.548232 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:43.547909    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:44.547663 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:44.547627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:44.547823 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:44.547634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:44.547823 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:44.547771    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:44.548223 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:44.547844    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:45.547733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:45.547698    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:45.547924 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:45.547844    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:46.547232 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:46.547194    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:46.547392 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:46.547243    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:46.547392 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:46.547340    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:46.547522 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:46.547472    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:47.547194 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:47.547145    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:47.547567 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:47.547266    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:48.317788 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:48.317741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:48.317992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:48.317800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:48.317992 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.317891    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:48.317992 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.317965    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:04.317949765 +0000 UTC m=+34.203922236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:15:48.317992 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.317900    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:48.318152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.318050    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:04.318031458 +0000 UTC m=+34.204003933 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:15:48.419085 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:48.419049    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:48.419275 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.419214    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:15:48.419275 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.419235    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:15:48.419275 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.419246    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:48.419443 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.419323    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:04.41930296 +0000 UTC m=+34.305275427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:15:48.547590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:48.547559    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:48.548039 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.547697    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:48.548039 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:48.547760    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:48.548039 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:48.547877    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:49.547181 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:49.547128    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:49.547352 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:49.547280    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:50.548441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:50.548405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:50.548895 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:50.548498    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:50.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:50.548557    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:50.548895 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:50.548691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:51.547049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:51.547031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:51.547137 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:51.547120    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:52.548448 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.548152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:52.548448 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.548246    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:52.549267 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:52.548462    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:52.549267 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:52.548575    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:52.665321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.665285    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln" event={"ID":"f22573dd-73e1-47ad-8bd4-0aac29bcb589","Type":"ContainerStarted","Data":"af2f6f4d5ba8d5c671dcfc46e77c18244041e63e7255efdea613559e0a2d778b"}
Sep 11 18:15:52.666860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.666834    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s7h6h" event={"ID":"44dd8471-2506-4cc9-9c44-9f9572550751","Type":"ContainerStarted","Data":"3b8879d1b79d3e524feaf39f97677b856e3fe95720412a96467a8ea6e0969266"}
Sep 11 18:15:52.668597 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.668574    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-n6pjd" event={"ID":"a948bb34-43b3-4f19-8cd0-0851b7f71882","Type":"ContainerStarted","Data":"c069e272e1bb2aae56cbeb756e0c54cffc2daf8f76d6ef890b9ac6270b151a1a"}
Sep 11 18:15:52.670469 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.670415    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlqt7" event={"ID":"fecc12ff-46ac-4ca5-8a23-c6b426446bb2","Type":"ContainerStarted","Data":"5844ba92ac0821c34b6fa55cf1955e2a76c6668a97a0cbdc46271d7c56bb8429"}
Sep 11 18:15:52.673787 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.673202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qdmqh" event={"ID":"2aeadb79-70da-4d8e-a32c-276592debddb","Type":"ContainerStarted","Data":"33977cbf9e254942196adbca945a14f8751f497aa71284c984bb46d711fcba46"}
Sep 11 18:15:52.678028 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678001    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"fe92c61527348f749d13b9c1176f3c35ec27665f021f21de082bb3d3e9a33bc1"}
Sep 11 18:15:52.678159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"d6a5e726d90c88e9f4fbf23eb8c6f3daf1158846ee20d73e272c1a0aa1fb0cb9"}
Sep 11 18:15:52.678159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678130    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"8e31464119f67241f823f0964e3c0df9c8ac4628444a93d30f59d5bc6f986def"}
Sep 11 18:15:52.678296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678193    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"5a324d032287ea8bfef500d558e71b713a4714e36823e4aa8a8b5553ecfc5ab8"}
Sep 11 18:15:52.678296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678240    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"680ef6d3eb3f42229f20681bebfb208851769b4206d999c0aeb5ffc76c79cf9c"}
Sep 11 18:15:52.678296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.678289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"ca213bd9a61ccbb00300473539ec711b99cfc1b22752d5a4824a0ee021a002f3"}
Sep 11 18:15:52.679620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.679597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-v8bhv" event={"ID":"060637b0-6735-4ce5-b082-6c943d98b6d2","Type":"ContainerStarted","Data":"cce2bbff89adae3ae6ba354c601002d7c831de7d40a8d2654a78f93743fef952"}
Sep 11 18:15:52.681243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.681219    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="8421abba56edd935c6942e3937155e7d325c59e896c77586a0dd088199d3942a" exitCode=0
Sep 11 18:15:52.681344 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.681250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"8421abba56edd935c6942e3937155e7d325c59e896c77586a0dd088199d3942a"}
Sep 11 18:15:52.696277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.696234    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s7h6h" podStartSLOduration=3.63919053 podStartE2EDuration="21.696222682s" totalImagesPullingTime="18.057032152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.69603008 +0000 UTC m=+22.582002571" watchObservedRunningTime="2026-09-11 18:15:52.696222682 +0000 UTC m=+22.582195171"
Sep 11 18:15:52.696369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.696352    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-148.ec2.internal" podStartSLOduration=21.696345932 podStartE2EDuration="21.696345932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:35.644004002 +0000 UTC m=+5.529976486" watchObservedRunningTime="2026-09-11 18:15:52.696345932 +0000 UTC m=+22.582318421"
Sep 11 18:15:52.715883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.715841    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-n6pjd" podStartSLOduration=4.655291721 podStartE2EDuration="22.715829259s" totalImagesPullingTime="18.060537538s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.715314082 +0000 UTC m=+22.601286571" watchObservedRunningTime="2026-09-11 18:15:52.715829259 +0000 UTC m=+22.601801748"
Sep 11 18:15:52.733544 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.733498    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-v8bhv" podStartSLOduration=4.573265542 podStartE2EDuration="22.733485169s" totalImagesPullingTime="18.160219627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.733394544 +0000 UTC m=+22.619367035" watchObservedRunningTime="2026-09-11 18:15:52.733485169 +0000 UTC m=+22.619457658"
Sep 11 18:15:52.778057 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.778011    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qdmqh" podStartSLOduration=12.230884239 podStartE2EDuration="22.777996193s" totalImagesPullingTime="10.547111954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.777551666 +0000 UTC m=+22.663524155" watchObservedRunningTime="2026-09-11 18:15:52.777996193 +0000 UTC m=+22.663968682"
Sep 11 18:15:52.795888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:52.795849    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nlqt7" podStartSLOduration=4.731913832 podStartE2EDuration="22.79583655s" totalImagesPullingTime="18.063922718s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:52.795416938 +0000 UTC m=+22.681389427" watchObservedRunningTime="2026-09-11 18:15:52.79583655 +0000 UTC m=+22.681809058"
Sep 11 18:15:53.074879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.074853    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 18:15:53.548011 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.547972    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:53.548260 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:53.548117    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:53.559924 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.559807    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T18:15:53.074870546Z","UUID":"f3f301cc-360c-437a-adf6-8597c6e6373f","Handler":null,"Name":"","Endpoint":""}
Sep 11 18:15:53.561433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.561412    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 18:15:53.561548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.561450    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 18:15:53.686005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.685972    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln" event={"ID":"f22573dd-73e1-47ad-8bd4-0aac29bcb589","Type":"ContainerStarted","Data":"34291993ac6310b8834b48d5a97d341ebf670b28939187818529bce949aae44c"}
Sep 11 18:15:53.688697 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:53.688670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-44bnm" event={"ID":"cc8b33b1-a528-4085-9efe-a4873d6278a8","Type":"ContainerStarted","Data":"7c40890e7cf3b16586d2e3058e9ac85857085c16ef6f5076499746aec9ad3f17"}
Sep 11 18:15:54.548192 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.548085    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:54.548192 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.548136    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:54.548410 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:54.548282    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:54.548471 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:54.548395    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:54.694003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.693887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln" event={"ID":"f22573dd-73e1-47ad-8bd4-0aac29bcb589","Type":"ContainerStarted","Data":"4a82d94000961946d7e463ce8b766c62fdecb82a7234a27d1dd22fa224f84e79"}
Sep 11 18:15:54.697851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.697818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"3ad1aa42d6fddc9f00c20a4068fed535e1e61e6195cbfe7ba7d7feb8b3411c2d"}
Sep 11 18:15:54.714391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.714330    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j9cln" podStartSLOduration=4.441785377 podStartE2EDuration="24.714311972s" totalImagesPullingTime="20.272526595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:54.714075631 +0000 UTC m=+24.600048119" watchObservedRunningTime="2026-09-11 18:15:54.714311972 +0000 UTC m=+24.600284463"
Sep 11 18:15:54.714708 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:54.714679    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-44bnm" podStartSLOduration=6.657598216 podStartE2EDuration="24.714654285s" totalImagesPullingTime="18.057056069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:53.705274951 +0000 UTC m=+23.591247441" watchObservedRunningTime="2026-09-11 18:15:54.714654285 +0000 UTC m=+24.600626776"
Sep 11 18:15:55.492360 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:55.492321    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:55.492968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:55.492946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:55.547614 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:55.547587    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:55.547792 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:55.547682    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:55.700827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:55.700798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:55.701290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:55.701270    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qdmqh"
Sep 11 18:15:56.547249 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:56.547221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:56.547405 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:56.547231    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:56.547405 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:56.547337    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:56.547483 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:56.547419    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:57.547555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.547384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:57.548226 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:57.547630    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:57.708234 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.708196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" event={"ID":"a6eb9f67-93e1-45cb-b5c7-c6ebff4dc73d","Type":"ContainerStarted","Data":"001352dfccf8df296a4e25d888622821489ef585d95d2402aabfb08043f5a59e"}
Sep 11 18:15:57.708580 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.708547    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:57.708749 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.708649    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:57.710351 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.710322    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="a6257b17f6109e5d61430b4032aa5a7e6988914bf21cb1bca83d5e681c194400" exitCode=0
Sep 11 18:15:57.710446 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.710377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"a6257b17f6109e5d61430b4032aa5a7e6988914bf21cb1bca83d5e681c194400"}
Sep 11 18:15:57.724125 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.724106    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:57.724230 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.724218    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:57.741892 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:57.741853    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-967q2" podStartSLOduration=9.633746345 podStartE2EDuration="27.741839524s" totalImagesPullingTime="18.108093179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:15:57.74132192 +0000 UTC m=+27.627294410" watchObservedRunningTime="2026-09-11 18:15:57.741839524 +0000 UTC m=+27.627812012"
Sep 11 18:15:58.548016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.547988    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:58.548482 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:58.548124    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:58.548547 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.548490    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:58.548615 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:58.548596    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:58.730457 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.730223    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:15:58.840400 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.840369    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7p4mx"]
Sep 11 18:15:58.840543 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.840524    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:15:58.840685 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:58.840626    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:15:58.842445 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.842416    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd2j8"]
Sep 11 18:15:58.842696 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.842679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:15:58.842829 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:58.842797    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:15:58.858906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.858880    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2x9w6"]
Sep 11 18:15:58.859000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:58.858990    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:15:58.859128 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:15:58.859105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:15:59.716611 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:59.716578    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="e10e4ba6a236280879b6cf00f3b810980881739422a6cf896e81a30eccc70da4" exitCode=0
Sep 11 18:15:59.716996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:15:59.716668    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"e10e4ba6a236280879b6cf00f3b810980881739422a6cf896e81a30eccc70da4"}
Sep 11 18:16:00.548644 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:00.548606    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:16:00.548816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:00.548653    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:00.548816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:00.548728    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:00.548816 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:00.548757    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:16:00.548816 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:00.548802    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:16:00.549012 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:00.548905    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:16:00.725316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:00.725278    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="f42700e5c8b40c6b72705b8f45cf68c97e01f2af520591f1666cbde3d757828a" exitCode=0
Sep 11 18:16:00.725674 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:00.725326    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"f42700e5c8b40c6b72705b8f45cf68c97e01f2af520591f1666cbde3d757828a"}
Sep 11 18:16:02.547381 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:02.547290    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:02.547381 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:02.547324    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:16:02.547887 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:02.547296    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:02.547887 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:02.547437    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kd2j8" podUID="0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6"
Sep 11 18:16:02.547887 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:02.547510    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7p4mx" podUID="049c6aea-3e27-4727-928b-c7cfea47f622"
Sep 11 18:16:02.547887 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:02.547608    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:16:04.337277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.337242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:04.337746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.337295    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:16:04.337746 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.337401    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:04.337746 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.337458    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.337440072 +0000 UTC m=+66.223412553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 18:16:04.337746 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.337402    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:04.337746 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.337529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret podName:0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.337513404 +0000 UTC m=+66.223485876 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret") pod "global-pull-secret-syncer-kd2j8" (UID: "0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6") : object "kube-system"/"original-pull-secret" not registered
Sep 11 18:16:04.436033 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.436004    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-148.ec2.internal" event="NodeReady"
Sep 11 18:16:04.436202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.436153    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 18:16:04.438245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.438224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:04.438425 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.438405    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 18:16:04.438472 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.438432    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 18:16:04.438472 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.438442    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nq4h2 for pod openshift-network-diagnostics/network-check-target-7p4mx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:04.438543 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.438494    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2 podName:049c6aea-3e27-4727-928b-c7cfea47f622 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.438475421 +0000 UTC m=+66.324447890 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nq4h2" (UniqueName: "kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2") pod "network-check-target-7p4mx" (UID: "049c6aea-3e27-4727-928b-c7cfea47f622") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 18:16:04.475337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.475296    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"]
Sep 11 18:16:04.538493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.538452    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"]
Sep 11 18:16:04.538670 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.538578    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.542313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.542288    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 18:16:04.542434 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.542299    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 18:16:04.542434 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.542340    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 18:16:04.542679 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.542648    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-rfr65\""
Sep 11 18:16:04.543083 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.543061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 18:16:04.553073 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.553052    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.555997 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.555966    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 18:16:04.556090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.556017    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 18:16:04.556090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.556017    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 18:16:04.556090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.556062    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 18:16:04.557164 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.557140    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588dcfb79-hsqjt"]
Sep 11 18:16:04.579098 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.579076    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"]
Sep 11 18:16:04.579260 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.579152    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:04.579260 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.579161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:04.579260 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.579164    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.579440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.579421    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:16:04.581750 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.581733    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:04.581889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.581868    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-76gd9\""
Sep 11 18:16:04.582048 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582027    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:04.582281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582265    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p9p5l\""
Sep 11 18:16:04.582524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582510    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 18:16:04.582599 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bn27r\""
Sep 11 18:16:04.582671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582651    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:04.582775 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582675    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 18:16:04.582775 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.582694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 18:16:04.583035 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.583013    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:04.588832 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.588815    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 18:16:04.596958 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.596939    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"]
Sep 11 18:16:04.597055 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.596970    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"]
Sep 11 18:16:04.597055 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.596993    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-98h96"]
Sep 11 18:16:04.597149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.597054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.599492 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.599476    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 18:16:04.619199 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.619137    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588dcfb79-hsqjt"]
Sep 11 18:16:04.619300 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.619202    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wlzz5"]
Sep 11 18:16:04.619419 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.619384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.623567 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.623427    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 18:16:04.623567 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.623567    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 18:16:04.623737 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.623717    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bf6d6\""
Sep 11 18:16:04.638944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.638923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82437cc-d2c2-4bac-95c6-1490f5c32c46-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.639049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.638952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4de13a-4266-41cd-9d70-d6036679f734-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.639049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.638970    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.639049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.639005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.639049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.639031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.639270 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.639066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gccm6\" (UniqueName: \"kubernetes.io/projected/e82437cc-d2c2-4bac-95c6-1490f5c32c46-kube-api-access-gccm6\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.639270 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.639142    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkn7m\" (UniqueName: \"kubernetes.io/projected/7d4de13a-4266-41cd-9d70-d6036679f734-kube-api-access-mkn7m\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.639357 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.639271    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.642883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.642860    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98h96"]
Sep 11 18:16:04.643000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.642904    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"]
Sep 11 18:16:04.643000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.642925    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlzz5"]
Sep 11 18:16:04.643000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.642937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.645929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.645911    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gb2xv\""
Sep 11 18:16:04.646439 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.646420    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 18:16:04.646542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.646518    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 18:16:04.646617 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.646554    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 18:16:04.739879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.739812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.739879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.739863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740057 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.739884    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng2rw\" (UniqueName: \"kubernetes.io/projected/cc1f7966-16e2-47a2-98c5-084115142030-kube-api-access-ng2rw\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.740057 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740003    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82437cc-d2c2-4bac-95c6-1490f5c32c46-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740057 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740042    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/415f29be-429b-4e63-8ad4-0f801c88799e-klusterlet-config\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.740197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4de13a-4266-41cd-9d70-d6036679f734-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.740197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b8c5e6-beb4-4aca-a918-284fd81f9210-tmp-dir\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.740350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740341    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740367    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740404    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6sj8\" (UniqueName: \"kubernetes.io/projected/415f29be-429b-4e63-8ad4-0f801c88799e-kube-api-access-s6sj8\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740435    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02b8c5e6-beb4-4aca-a918-284fd81f9210-config-volume\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740459    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9qqx\" (UniqueName: \"kubernetes.io/projected/02b8c5e6-beb4-4aca-a918-284fd81f9210-kube-api-access-w9qqx\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740506    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqtrd\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e82437cc-d2c2-4bac-95c6-1490f5c32c46-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gccm6\" (UniqueName: \"kubernetes.io/projected/e82437cc-d2c2-4bac-95c6-1490f5c32c46-kube-api-access-gccm6\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1f7966-16e2-47a2-98c5-084115142030-cert\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkn7m\" (UniqueName: \"kubernetes.io/projected/7d4de13a-4266-41cd-9d70-d6036679f734-kube-api-access-mkn7m\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.740889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740885    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f29be-429b-4e63-8ad4-0f801c88799e-tmp\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.741248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.740905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.743953 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.743927    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-ca\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.744131 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.744111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.744271 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.744218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.744271 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.744255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e82437cc-d2c2-4bac-95c6-1490f5c32c46-hub\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.744353 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.744280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7d4de13a-4266-41cd-9d70-d6036679f734-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.748867 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.748839    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkn7m\" (UniqueName: \"kubernetes.io/projected/7d4de13a-4266-41cd-9d70-d6036679f734-kube-api-access-mkn7m\") pod \"managed-serviceaccount-addon-agent-78678dd8c4-vgjmr\" (UID: \"7d4de13a-4266-41cd-9d70-d6036679f734\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.748979 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.748962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gccm6\" (UniqueName: \"kubernetes.io/projected/e82437cc-d2c2-4bac-95c6-1490f5c32c46-kube-api-access-gccm6\") pod \"cluster-proxy-proxy-agent-8665d5996-fcrlz\" (UID: \"e82437cc-d2c2-4bac-95c6-1490f5c32c46\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.841944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.841909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b8c5e6-beb4-4aca-a918-284fd81f9210-tmp-dir\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.841944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.841946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.842213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842108    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6sj8\" (UniqueName: \"kubernetes.io/projected/415f29be-429b-4e63-8ad4-0f801c88799e-kube-api-access-s6sj8\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.842213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02b8c5e6-beb4-4aca-a918-284fd81f9210-config-volume\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.842213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9qqx\" (UniqueName: \"kubernetes.io/projected/02b8c5e6-beb4-4aca-a918-284fd81f9210-kube-api-access-w9qqx\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.842213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.842229    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02b8c5e6-beb4-4aca-a918-284fd81f9210-tmp-dir\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.842254    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.842268    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.842307    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:05.342286097 +0000 UTC m=+35.228258578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:04.842340    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:05.342329286 +0000 UTC m=+35.228301759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842354    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqtrd\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842394    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1f7966-16e2-47a2-98c5-084115142030-cert\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842465    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f29be-429b-4e63-8ad4-0f801c88799e-tmp\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842491    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842546    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842581    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng2rw\" (UniqueName: \"kubernetes.io/projected/cc1f7966-16e2-47a2-98c5-084115142030-kube-api-access-ng2rw\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842734    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/02b8c5e6-beb4-4aca-a918-284fd81f9210-config-volume\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/415f29be-429b-4e63-8ad4-0f801c88799e-klusterlet-config\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.842992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.842871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/415f29be-429b-4e63-8ad4-0f801c88799e-tmp\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.843340 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.843004    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.843340 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.843045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.844874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.844846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.844982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.844941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cc1f7966-16e2-47a2-98c5-084115142030-cert\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.845067 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.845049    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/415f29be-429b-4e63-8ad4-0f801c88799e-klusterlet-config\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.845216 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.845197    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.851975 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.851943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng2rw\" (UniqueName: \"kubernetes.io/projected/cc1f7966-16e2-47a2-98c5-084115142030-kube-api-access-ng2rw\") pod \"ingress-canary-wlzz5\" (UID: \"cc1f7966-16e2-47a2-98c5-084115142030\") " pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:04.852475 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.852450    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqtrd\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.852562 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.852488    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6sj8\" (UniqueName: \"kubernetes.io/projected/415f29be-429b-4e63-8ad4-0f801c88799e-kube-api-access-s6sj8\") pod \"klusterlet-addon-workmgr-56fb568d4c-rnxh4\" (UID: \"415f29be-429b-4e63-8ad4-0f801c88799e\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.853516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.853495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:04.853657 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.853638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9qqx\" (UniqueName: \"kubernetes.io/projected/02b8c5e6-beb4-4aca-a918-284fd81f9210-kube-api-access-w9qqx\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:04.858431 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.858413    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:16:04.867597 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.867580    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:16:04.922088 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.921914    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:04.952602 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:04.952518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wlzz5"
Sep 11 18:16:05.176284 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.176256    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"]
Sep 11 18:16:05.180430 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:05.179936    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d4de13a_4266_41cd_9d70_d6036679f734.slice/crio-8545fbc23c04bce4dc4ac845080562a7fd829eb97c5590bb319f58fe3a3c61a5 WatchSource:0}: Error finding container 8545fbc23c04bce4dc4ac845080562a7fd829eb97c5590bb319f58fe3a3c61a5: Status 404 returned error can't find the container with id 8545fbc23c04bce4dc4ac845080562a7fd829eb97c5590bb319f58fe3a3c61a5
Sep 11 18:16:05.181015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.180992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"]
Sep 11 18:16:05.183759 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:05.183724    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode82437cc_d2c2_4bac_95c6_1490f5c32c46.slice/crio-0ef3b7ec2078a38e66465b2bf9b1ed2fc39e903ce55b581104943770edb808dd WatchSource:0}: Error finding container 0ef3b7ec2078a38e66465b2bf9b1ed2fc39e903ce55b581104943770edb808dd: Status 404 returned error can't find the container with id 0ef3b7ec2078a38e66465b2bf9b1ed2fc39e903ce55b581104943770edb808dd
Sep 11 18:16:05.215860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.215833    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"]
Sep 11 18:16:05.219050 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:05.219023    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f29be_429b_4e63_8ad4_0f801c88799e.slice/crio-798f8ab04d3ad3245c6fd22682d702cb6ce008942bf121cd57c984b3421ab452 WatchSource:0}: Error finding container 798f8ab04d3ad3245c6fd22682d702cb6ce008942bf121cd57c984b3421ab452: Status 404 returned error can't find the container with id 798f8ab04d3ad3245c6fd22682d702cb6ce008942bf121cd57c984b3421ab452
Sep 11 18:16:05.244985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.244937    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wlzz5"]
Sep 11 18:16:05.251449 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:05.251424    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1f7966_16e2_47a2_98c5_084115142030.slice/crio-b35f159f8c048b3333e34006e15ad7f7a2d8bc53ed4e86b71a8931630421ae61 WatchSource:0}: Error finding container b35f159f8c048b3333e34006e15ad7f7a2d8bc53ed4e86b71a8931630421ae61: Status 404 returned error can't find the container with id b35f159f8c048b3333e34006e15ad7f7a2d8bc53ed4e86b71a8931630421ae61
Sep 11 18:16:05.346631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.346600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.346642    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:05.346745    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:05.346809    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:06.346795013 +0000 UTC m=+36.232767483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:05.346749    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:05.346851    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:05.347213 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:05.346887    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:06.34687625 +0000 UTC m=+36.232848721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:05.684866 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.684833    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qdmqh_2aeadb79-70da-4d8e-a32c-276592debddb/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:05.738727 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.738688    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4" event={"ID":"415f29be-429b-4e63-8ad4-0f801c88799e","Type":"ContainerStarted","Data":"798f8ab04d3ad3245c6fd22682d702cb6ce008942bf121cd57c984b3421ab452"}
Sep 11 18:16:05.739819 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.739785    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerStarted","Data":"0ef3b7ec2078a38e66465b2bf9b1ed2fc39e903ce55b581104943770edb808dd"}
Sep 11 18:16:05.740769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.740738    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr" event={"ID":"7d4de13a-4266-41cd-9d70-d6036679f734","Type":"ContainerStarted","Data":"8545fbc23c04bce4dc4ac845080562a7fd829eb97c5590bb319f58fe3a3c61a5"}
Sep 11 18:16:05.741840 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.741812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlzz5" event={"ID":"cc1f7966-16e2-47a2-98c5-084115142030","Type":"ContainerStarted","Data":"b35f159f8c048b3333e34006e15ad7f7a2d8bc53ed4e86b71a8931630421ae61"}
Sep 11 18:16:05.744668 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.744644    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="2d0541b34ceab2d68ff32793485626ad39253533c0203c6b5df9593243e42b92" exitCode=0
Sep 11 18:16:05.744781 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.744675    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"2d0541b34ceab2d68ff32793485626ad39253533c0203c6b5df9593243e42b92"}
Sep 11 18:16:05.780578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:05.780550    2582 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qdmqh_2aeadb79-70da-4d8e-a32c-276592debddb/konnectivity-agent/0.log" limit=1024
Sep 11 18:16:06.356268 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:06.356212    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:06.356919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:06.356277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:06.356919 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:06.356464    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:06.356919 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:06.356480    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:06.356919 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:06.356539    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:08.356521753 +0000 UTC m=+38.242494239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:06.357217 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:06.356936    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:06.357217 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:06.356988    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:08.356972819 +0000 UTC m=+38.242945301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:06.766421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:06.766379    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4965d6-e8d5-4227-a78f-d28d3bc2278d" containerID="dfa1b07ddacab4e5bf63825a4e3e569f5da1c94cb80b88ec7d344a12800c1fa5" exitCode=0
Sep 11 18:16:06.766601 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:06.766437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerDied","Data":"dfa1b07ddacab4e5bf63825a4e3e569f5da1c94cb80b88ec7d344a12800c1fa5"}
Sep 11 18:16:07.777667 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:07.777412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" event={"ID":"fa4965d6-e8d5-4227-a78f-d28d3bc2278d","Type":"ContainerStarted","Data":"34c888fea60ca0811251bd1a8a1c2fac02db4b03f96578f6f80e8ca057ffd2e8"}
Sep 11 18:16:07.803846 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:07.803772    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jlbkg" podStartSLOduration=9.422188096 podStartE2EDuration="37.803750299s" totalImagesPullingTime="28.381562203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:07.803619836 +0000 UTC m=+37.689592328" watchObservedRunningTime="2026-09-11 18:16:07.803750299 +0000 UTC m=+37.689722789"
Sep 11 18:16:08.378250 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:08.378208    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:08.378412 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:08.378276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:08.378476 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:08.378407    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:08.378476 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:08.378441    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:08.378476 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:08.378456    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:08.378628 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:08.378501    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:12.378479206 +0000 UTC m=+42.264451688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:08.378628 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:08.378522    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:12.378512099 +0000 UTC m=+42.264484573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:12.414688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:12.414650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:12.414707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:12.414822    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:12.414864    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:12.414878    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:12.414899    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:20.414877284 +0000 UTC m=+50.300849754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:12.415123 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:12.414928    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:20.414911821 +0000 UTC m=+50.300884293 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:14.798459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.798419    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4" event={"ID":"415f29be-429b-4e63-8ad4-0f801c88799e","Type":"ContainerStarted","Data":"f8cc37113e5d70ed1e4720e609620df5c6b13009d007ded7b220cbe597a55584"}
Sep 11 18:16:14.798878 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.798626    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:14.800622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.800601    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:16:14.800796 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.800770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerStarted","Data":"ca908762f0c80dd48b2d392b9d5c8101ad4aac746a71cf08e93ab437eae207dd"}
Sep 11 18:16:14.800865 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.800812    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerStarted","Data":"456f80444a6014f47a4d2e72e1d48984b8ce448a170df7d51acf615f33b6d641"}
Sep 11 18:16:14.800865 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.800830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerStarted","Data":"a003201bb30031bd32cdbb1ea47bc9db32065cd9ccd28f73ae38cdf26de5e7cc"}
Sep 11 18:16:14.802155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.802133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr" event={"ID":"7d4de13a-4266-41cd-9d70-d6036679f734","Type":"ContainerStarted","Data":"476c18d10d2aea9385424bdc3a645cc0d23bb805e73532b23151df264ad6807d"}
Sep 11 18:16:14.803466 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.803445    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wlzz5" event={"ID":"cc1f7966-16e2-47a2-98c5-084115142030","Type":"ContainerStarted","Data":"b4d17f2f05869c39c64e2e13f055df8da8a50834ee65160c7a4a8bbe4c5c3c2f"}
Sep 11 18:16:14.815589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.815546    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4" podStartSLOduration=20.24544811 podStartE2EDuration="28.815534187s" totalImagesPullingTime="8.570086077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:14.814388114 +0000 UTC m=+44.700360604" watchObservedRunningTime="2026-09-11 18:16:14.815534187 +0000 UTC m=+44.701506669"
Sep 11 18:16:14.853624 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.853568    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" podStartSLOduration=20.232910716 podStartE2EDuration="28.853551153s" totalImagesPullingTime="8.620640437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:14.853162166 +0000 UTC m=+44.739134655" watchObservedRunningTime="2026-09-11 18:16:14.853551153 +0000 UTC m=+44.739523645"
Sep 11 18:16:14.869285 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.869246    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wlzz5" podStartSLOduration=2.331437406 podStartE2EDuration="10.869233976s" totalImagesPullingTime="8.53779657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:14.868479214 +0000 UTC m=+44.754451703" watchObservedRunningTime="2026-09-11 18:16:14.869233976 +0000 UTC m=+44.755206465"
Sep 11 18:16:14.886802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:14.886749    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr" podStartSLOduration=20.278004646 podStartE2EDuration="28.886736206s" totalImagesPullingTime="8.60873156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:14.886438713 +0000 UTC m=+44.772411200" watchObservedRunningTime="2026-09-11 18:16:14.886736206 +0000 UTC m=+44.772708696"
Sep 11 18:16:20.477071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:20.477030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:20.477071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:20.477077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:20.477552 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:20.477191    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:20.477552 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:20.477200    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:20.477552 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:20.477270    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.477256226 +0000 UTC m=+66.363228693 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:20.477552 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:20.477206    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:20.477552 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:20.477354    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:16:36.47733895 +0000 UTC m=+66.363311417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:29.727441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:29.727411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-967q2"
Sep 11 18:16:36.392183 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.392114    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:36.392183 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.392187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:16:36.395900 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.395882    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 18:16:36.396180 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.396155    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 18:16:36.403021 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.403004    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:16:36.403109 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.403097    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:40.403072518 +0000 UTC m=+130.289044990 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : secret "metrics-daemon-secret" not found
Sep 11 18:16:36.406431 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.406415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6-original-pull-secret\") pod \"global-pull-secret-syncer-kd2j8\" (UID: \"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6\") " pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:36.492908 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.492866    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:36.492908 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.492914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.492990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.493018    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.493079    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.493097    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.493087    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:08.493068638 +0000 UTC m=+98.379041105 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:16:36.493152 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:16:36.493146    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:17:08.493134428 +0000 UTC m=+98.379106895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:16:36.495525 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.495507    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 18:16:36.505248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.505229    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 18:16:36.516223 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.516195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4h2\" (UniqueName: \"kubernetes.io/projected/049c6aea-3e27-4727-928b-c7cfea47f622-kube-api-access-nq4h2\") pod \"network-check-target-7p4mx\" (UID: \"049c6aea-3e27-4727-928b-c7cfea47f622\") " pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:36.697819 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.697790    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-76gd9\""
Sep 11 18:16:36.700705 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.700690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kd2j8"
Sep 11 18:16:36.705292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.705275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:36.971486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.971415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kd2j8"]
Sep 11 18:16:36.974365 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:36.974338    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7p4mx"]
Sep 11 18:16:36.974872 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:36.974848    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0922c1e3_dc51_4b8c_ae1e_ac9af86c54e6.slice/crio-329e89855cd889652e55a50625386f8f092a7f6c0d6ff2db85ad0cf37d0b2820 WatchSource:0}: Error finding container 329e89855cd889652e55a50625386f8f092a7f6c0d6ff2db85ad0cf37d0b2820: Status 404 returned error can't find the container with id 329e89855cd889652e55a50625386f8f092a7f6c0d6ff2db85ad0cf37d0b2820
Sep 11 18:16:36.976995 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:16:36.976972    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049c6aea_3e27_4727_928b_c7cfea47f622.slice/crio-3f83060b745badfdbf564f8dfcdfc995f4a85a451dd2cce80d61491569c32727 WatchSource:0}: Error finding container 3f83060b745badfdbf564f8dfcdfc995f4a85a451dd2cce80d61491569c32727: Status 404 returned error can't find the container with id 3f83060b745badfdbf564f8dfcdfc995f4a85a451dd2cce80d61491569c32727
Sep 11 18:16:37.874189 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:37.874133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7p4mx" event={"ID":"049c6aea-3e27-4727-928b-c7cfea47f622","Type":"ContainerStarted","Data":"3f83060b745badfdbf564f8dfcdfc995f4a85a451dd2cce80d61491569c32727"}
Sep 11 18:16:37.875662 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:37.875623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd2j8" event={"ID":"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6","Type":"ContainerStarted","Data":"329e89855cd889652e55a50625386f8f092a7f6c0d6ff2db85ad0cf37d0b2820"}
Sep 11 18:16:42.894322 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:42.894277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7p4mx" event={"ID":"049c6aea-3e27-4727-928b-c7cfea47f622","Type":"ContainerStarted","Data":"52056c4829e6e369e729ce4a96fa41a4202468954d5a58a80844066c4065f2fe"}
Sep 11 18:16:42.894793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:42.894405    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:16:42.895835 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:42.895813    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kd2j8" event={"ID":"0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6","Type":"ContainerStarted","Data":"f58fd820a547289b4f235e723dc99cad341c0168c332624693d56916aa623de5"}
Sep 11 18:16:42.944296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:42.944247    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7p4mx" podStartSLOduration=67.623252504 podStartE2EDuration="1m12.944231041s" totalImagesPullingTime="5.320978537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:42.919023554 +0000 UTC m=+72.804996044" watchObservedRunningTime="2026-09-11 18:16:42.944231041 +0000 UTC m=+72.830203526"
Sep 11 18:16:42.944595 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:16:42.944575    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kd2j8" podStartSLOduration=67.61681552 podStartE2EDuration="1m12.944567544s" totalImagesPullingTime="5.327752024s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:16:42.943660843 +0000 UTC m=+72.829633335" watchObservedRunningTime="2026-09-11 18:16:42.944567544 +0000 UTC m=+72.830540069"
Sep 11 18:17:08.543847 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:17:08.543812    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:17:08.543856    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:08.543970    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:08.544043    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:12.544028219 +0000 UTC m=+162.430000691 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:08.543974    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:08.544076    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:17:08.544228 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:08.544125    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:12.544113792 +0000 UTC m=+162.430086262 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:17:13.901869 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:17:13.901838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7p4mx"
Sep 11 18:17:40.472995 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:17:40.472953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:17:40.473439 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:40.473123    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 18:17:40.473439 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:17:40.473221    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs podName:972c3842-6619-4463-802a-7a7ca6e6db54 nodeName:}" failed. No retries permitted until 2026-09-11 18:19:42.473203774 +0000 UTC m=+252.359176245 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs") pod "network-metrics-daemon-2x9w6" (UID: "972c3842-6619-4463-802a-7a7ca6e6db54") : secret "metrics-daemon-secret" not found
Sep 11 18:18:05.426704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.426674    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6dd9n"]
Sep 11 18:18:05.429018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.428937    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.432033 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.432010    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 18:18:05.432033 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.432015    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 18:18:05.432229 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.432051    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 18:18:05.432296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.432280    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 18:18:05.432809 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.432775    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wf5gj\""
Sep 11 18:18:05.444765 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.444744    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dd9n"]
Sep 11 18:18:05.556685 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.556653    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-crio-socket\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.556852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.556694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.556852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.556754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-data-volume\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.556852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.556771    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqdb6\" (UniqueName: \"kubernetes.io/projected/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-api-access-tqdb6\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.556852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.556808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657367    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-crio-socket\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657446    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-data-volume\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657470    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqdb6\" (UniqueName: \"kubernetes.io/projected/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-api-access-tqdb6\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-crio-socket\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.657575 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:05.657533    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:05.657792 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:05.657623    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:06.157607689 +0000 UTC m=+156.043580158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:05.657792 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:05.657622    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.657792 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:05.657688    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:06.157668768 +0000 UTC m=+156.043641250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:05.657792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.657775    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-data-volume\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:05.667464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:05.667424    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqdb6\" (UniqueName: \"kubernetes.io/projected/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-api-access-tqdb6\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:06.161127 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:06.161086    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:06.161313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:06.161165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:06.161313 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:06.161231    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:06.161313 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:06.161275    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:06.161313 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:06.161297    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:07.16128208 +0000 UTC m=+157.047254547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:06.161441 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:06.161325    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:07.161309347 +0000 UTC m=+157.047281814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.169400 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:07.169361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:07.169764 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:07.169417    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:07.169764 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.169508    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.169764 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.169537    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:07.169764 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.169563    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:09.169548483 +0000 UTC m=+159.055520950 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:07.169764 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.169584    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:09.169572522 +0000 UTC m=+159.055544989 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:07.607956 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.607865    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-2x9w6" podUID="972c3842-6619-4463-802a-7a7ca6e6db54"
Sep 11 18:18:07.616031 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.616007    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt" podUID="c2a63f36-d831-4c96-81bf-760ea614a4c8"
Sep 11 18:18:07.633341 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:07.633314    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-98h96" podUID="02b8c5e6-beb4-4aca-a918-284fd81f9210"
Sep 11 18:18:08.146513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:08.146482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:18:08.146513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:08.146512    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98h96"
Sep 11 18:18:08.392678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:08.392653    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7h6h_44dd8471-2506-4cc9-9c44-9f9572550751/dns-node-resolver/0.log"
Sep 11 18:18:09.187334 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:09.187291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:09.187541 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:09.187381    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:09.187541 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:09.187430    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:09.187541 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:09.187486    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:09.187541 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:09.187526    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.187503549 +0000 UTC m=+163.073476033 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:09.187761 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:09.187546    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:13.187537272 +0000 UTC m=+163.073509743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:09.193049 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:09.193032    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6pjd_a948bb34-43b3-4f19-8cd0-0851b7f71882/node-ca/0.log"
Sep 11 18:18:10.197593 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:10.197560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlzz5_cc1f7966-16e2-47a2-98c5-084115142030/serve-healthcheck-canary/0.log"
Sep 11 18:18:12.615393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:12.615364    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:18:12.615393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:12.615398    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") pod \"image-registry-588dcfb79-hsqjt\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") " pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:18:12.615801 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:12.615524    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 18:18:12.615801 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:12.615601    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls podName:02b8c5e6-beb4-4aca-a918-284fd81f9210 nodeName:}" failed. No retries permitted until 2026-09-11 18:20:14.615585356 +0000 UTC m=+284.501557839 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls") pod "dns-default-98h96" (UID: "02b8c5e6-beb4-4aca-a918-284fd81f9210") : secret "dns-default-metrics-tls" not found
Sep 11 18:18:12.615801 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:12.615527    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 18:18:12.615801 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:12.615625    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-588dcfb79-hsqjt: secret "image-registry-tls" not found
Sep 11 18:18:12.615801 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:12.615675    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls podName:c2a63f36-d831-4c96-81bf-760ea614a4c8 nodeName:}" failed. No retries permitted until 2026-09-11 18:20:14.61566446 +0000 UTC m=+284.501636926 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls") pod "image-registry-588dcfb79-hsqjt" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8") : secret "image-registry-tls" not found
Sep 11 18:18:13.220657 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:13.220619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:13.220842 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:13.220705    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:13.220842 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:13.220745    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 18:18:13.220842 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:13.220799    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:13.220842 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:13.220806    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.220791737 +0000 UTC m=+171.106764204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : configmap "kube-rbac-proxy" not found
Sep 11 18:18:13.220842 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:13.220838    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:21.22082767 +0000 UTC m=+171.106800150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:14.166365 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.166331    2582 generic.go:356] "Generic (PLEG): container finished" podID="415f29be-429b-4e63-8ad4-0f801c88799e" containerID="f8cc37113e5d70ed1e4720e609620df5c6b13009d007ded7b220cbe597a55584" exitCode=1
Sep 11 18:18:14.166821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.166413    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4" event={"ID":"415f29be-429b-4e63-8ad4-0f801c88799e","Type":"ContainerDied","Data":"f8cc37113e5d70ed1e4720e609620df5c6b13009d007ded7b220cbe597a55584"}
Sep 11 18:18:14.166821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.166807    2582 scope.go:117] "RemoveContainer" containerID="f8cc37113e5d70ed1e4720e609620df5c6b13009d007ded7b220cbe597a55584"
Sep 11 18:18:14.167903 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.167876    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d4de13a-4266-41cd-9d70-d6036679f734" containerID="476c18d10d2aea9385424bdc3a645cc0d23bb805e73532b23151df264ad6807d" exitCode=255
Sep 11 18:18:14.168018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.167927    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr" event={"ID":"7d4de13a-4266-41cd-9d70-d6036679f734","Type":"ContainerDied","Data":"476c18d10d2aea9385424bdc3a645cc0d23bb805e73532b23151df264ad6807d"}
Sep 11 18:18:14.168184 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.168155    2582 scope.go:117] "RemoveContainer" containerID="476c18d10d2aea9385424bdc3a645cc0d23bb805e73532b23151df264ad6807d"
Sep 11 18:18:14.799512 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.799485    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:18:14.858738 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.858707    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr"
Sep 11 18:18:14.922551 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:14.922517    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:18:15.172748 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:15.172720    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78678dd8c4-vgjmr" event={"ID":"7d4de13a-4266-41cd-9d70-d6036679f734","Type":"ContainerStarted","Data":"a98b94ea485789d54900e7f78159713d9deb4df2a211d166092fa7ddb592076d"}
Sep 11 18:18:15.174328 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:15.174305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4" event={"ID":"415f29be-429b-4e63-8ad4-0f801c88799e","Type":"ContainerStarted","Data":"d2eba064ae48cbf306d56ee1659e922ab3949fcacee9a6cb8b420870f1935f53"}
Sep 11 18:18:15.174509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:15.174493    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:18:15.175971 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:15.175953    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-56fb568d4c-rnxh4"
Sep 11 18:18:21.287845 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:21.287803    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:21.288256 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:21.287870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:21.288256 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:21.287963    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:21.288256 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:21.288026    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls podName:d7bc0426-d3ab-4adc-9160-b6c1bb0fd824 nodeName:}" failed. No retries permitted until 2026-09-11 18:18:37.288009965 +0000 UTC m=+187.173982448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6dd9n" (UID: "d7bc0426-d3ab-4adc-9160-b6c1bb0fd824") : secret "insights-runtime-extractor-tls" not found
Sep 11 18:18:21.288432 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:21.288415    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:22.547653 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:22.547618    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:18:34.868797 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:34.868699    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" podUID="e82437cc-d2c2-4bac-95c6-1490f5c32c46" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:18:37.312369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:37.312303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:37.314608 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:37.314584    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d7bc0426-d3ab-4adc-9160-b6c1bb0fd824-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6dd9n\" (UID: \"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824\") " pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:37.540497 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:37.540460    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wf5gj\""
Sep 11 18:18:37.548547 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:37.548523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6dd9n"
Sep 11 18:18:37.817241 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:37.817206    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6dd9n"]
Sep 11 18:18:37.820603 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:18:37.820567    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bc0426_d3ab_4adc_9160_b6c1bb0fd824.slice/crio-954ee3e112cdbc8c4102dca0b33b85c7c6422c600c46dd101603b28766d1b5a5 WatchSource:0}: Error finding container 954ee3e112cdbc8c4102dca0b33b85c7c6422c600c46dd101603b28766d1b5a5: Status 404 returned error can't find the container with id 954ee3e112cdbc8c4102dca0b33b85c7c6422c600c46dd101603b28766d1b5a5
Sep 11 18:18:38.239862 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:38.239829    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dd9n" event={"ID":"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824","Type":"ContainerStarted","Data":"9aeb3d48fa9a37333df38d4dc53b55cec99950561fe20b34e03d163083988b61"}
Sep 11 18:18:38.240022 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:38.239889    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dd9n" event={"ID":"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824","Type":"ContainerStarted","Data":"954ee3e112cdbc8c4102dca0b33b85c7c6422c600c46dd101603b28766d1b5a5"}
Sep 11 18:18:39.244995 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:39.244962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dd9n" event={"ID":"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824","Type":"ContainerStarted","Data":"a416eb7d89377799800297563e120a77b4ea132c47d687eb8470822b9957c512"}
Sep 11 18:18:40.249651 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:40.249618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6dd9n" event={"ID":"d7bc0426-d3ab-4adc-9160-b6c1bb0fd824","Type":"ContainerStarted","Data":"053c67bcaf7bb53f96a0a4d97d6251a038946038e15fda23f7cd84754a2c3015"}
Sep 11 18:18:40.268688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:40.268642    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6dd9n" podStartSLOduration=33.437775184 podStartE2EDuration="35.268626955s" totalImagesPullingTime="1.830851771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:40.26789105 +0000 UTC m=+190.153863541" watchObservedRunningTime="2026-09-11 18:18:40.268626955 +0000 UTC m=+190.154599443"
Sep 11 18:18:41.364549 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.364514    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7z5vk"]
Sep 11 18:18:41.366568 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.366553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.369738 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.369717    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 18:18:41.370674 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370641    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-962fk\""
Sep 11 18:18:41.370790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370718    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 18:18:41.370790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370727    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 18:18:41.370790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370777    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 18:18:41.370946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370823    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 18:18:41.370946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.370650    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 18:18:41.447827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-sys\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.447827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-metrics-client-ca\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-textfile\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447945    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-tls\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.447984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-root\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.448071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-wtmp\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.448105    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.448142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.448125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s55jm\" (UniqueName: \"kubernetes.io/projected/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-kube-api-access-s55jm\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548395    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-textfile\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-tls\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-root\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-wtmp\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548553    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s55jm\" (UniqueName: \"kubernetes.io/projected/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-kube-api-access-s55jm\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-sys\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548601    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-root\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-sys\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-metrics-client-ca\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-wtmp\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548772    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.548895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.548781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-textfile\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.549247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.549228    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-metrics-client-ca\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.549329 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.549309    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-accelerators-collector-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.550785 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.550770    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-tls\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.550890 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.550875    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.561877 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.561850    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s55jm\" (UniqueName: \"kubernetes.io/projected/5e63bfa1-1ee1-428d-a490-90ab6ebac60a-kube-api-access-s55jm\") pod \"node-exporter-7z5vk\" (UID: \"5e63bfa1-1ee1-428d-a490-90ab6ebac60a\") " pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.675735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:41.675706    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7z5vk"
Sep 11 18:18:41.684733 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:18:41.684707    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e63bfa1_1ee1_428d_a490_90ab6ebac60a.slice/crio-9d0ac9b2880d03758f0783bc957a248e340ee1fb6794d31095a05b45c6a0efb2 WatchSource:0}: Error finding container 9d0ac9b2880d03758f0783bc957a248e340ee1fb6794d31095a05b45c6a0efb2: Status 404 returned error can't find the container with id 9d0ac9b2880d03758f0783bc957a248e340ee1fb6794d31095a05b45c6a0efb2
Sep 11 18:18:42.260627 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:42.260583    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z5vk" event={"ID":"5e63bfa1-1ee1-428d-a490-90ab6ebac60a","Type":"ContainerStarted","Data":"9d0ac9b2880d03758f0783bc957a248e340ee1fb6794d31095a05b45c6a0efb2"}
Sep 11 18:18:43.265971 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:43.265933    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e63bfa1-1ee1-428d-a490-90ab6ebac60a" containerID="46e522596a7eb47330067bcb89b9c1e79bb242d48d98f2a70b7d2ce1cf82bdb4" exitCode=0
Sep 11 18:18:43.266374 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:43.265979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z5vk" event={"ID":"5e63bfa1-1ee1-428d-a490-90ab6ebac60a","Type":"ContainerDied","Data":"46e522596a7eb47330067bcb89b9c1e79bb242d48d98f2a70b7d2ce1cf82bdb4"}
Sep 11 18:18:44.271951 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:44.271920    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z5vk" event={"ID":"5e63bfa1-1ee1-428d-a490-90ab6ebac60a","Type":"ContainerStarted","Data":"6f5a8f459eda4e40b8151824ee4e5319a4957b9859f6d7435e2ec9e48c5353d2"}
Sep 11 18:18:44.272421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:44.271969    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7z5vk" event={"ID":"5e63bfa1-1ee1-428d-a490-90ab6ebac60a","Type":"ContainerStarted","Data":"85dde065b9be6617b0c17c8d228622cf866995bb2a17c389d5285224d4bc8775"}
Sep 11 18:18:44.296702 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:44.296647    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7z5vk" podStartSLOduration=2.501056533 podStartE2EDuration="3.296629391s" totalImagesPullingTime="733.741767ms" totalInitContainerRuntime="61.831091ms" isStatefulPod=true podCreationTimestamp="2026-09-11 18:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:18:44.294929506 +0000 UTC m=+194.180902032" watchObservedRunningTime="2026-09-11 18:18:44.296629391 +0000 UTC m=+194.182601880"
Sep 11 18:18:44.868919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:44.868872    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" podUID="e82437cc-d2c2-4bac-95c6-1490f5c32c46" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:18:52.555855 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:52.555815    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588dcfb79-hsqjt"]
Sep 11 18:18:52.556293 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:18:52.556083    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt" podUID="c2a63f36-d831-4c96-81bf-760ea614a4c8"
Sep 11 18:18:53.300292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.300261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:18:53.305359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.305325    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:18:53.435280 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435230    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435295    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435325    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435341    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqtrd\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435414    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435451    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration\") pod \"c2a63f36-d831-4c96-81bf-760ea614a4c8\" (UID: \"c2a63f36-d831-4c96-81bf-760ea614a4c8\") "
Sep 11 18:18:53.435734 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435540    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:18:53.435734 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435722    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c2a63f36-d831-4c96-81bf-760ea614a4c8-ca-trust-extracted\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.435908 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435858    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:53.435908 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.435869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:18:53.437767 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.437744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd" (OuterVolumeSpecName: "kube-api-access-vqtrd") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "kube-api-access-vqtrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:53.437880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.437800    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:18:53.437931 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.437900    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:53.437982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.437963    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c2a63f36-d831-4c96-81bf-760ea614a4c8" (UID: "c2a63f36-d831-4c96-81bf-760ea614a4c8"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:18:53.536343 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536309    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-certificates\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.536343 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536337    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-bound-sa-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.536524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536351    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-image-registry-private-configuration\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.536524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536364    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c2a63f36-d831-4c96-81bf-760ea614a4c8-installation-pull-secrets\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.536524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536375    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c2a63f36-d831-4c96-81bf-760ea614a4c8-trusted-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:53.536524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:53.536388    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqtrd\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-kube-api-access-vqtrd\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:54.303378 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.303348    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588dcfb79-hsqjt"
Sep 11 18:18:54.340183 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.340149    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-588dcfb79-hsqjt"]
Sep 11 18:18:54.344556 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.344535    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-588dcfb79-hsqjt"]
Sep 11 18:18:54.442781 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.442752    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c2a63f36-d831-4c96-81bf-760ea614a4c8-registry-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:18:54.551202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.551158    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2a63f36-d831-4c96-81bf-760ea614a4c8" path="/var/lib/kubelet/pods/c2a63f36-d831-4c96-81bf-760ea614a4c8/volumes"
Sep 11 18:18:54.868767 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.868727    2582 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" podUID="e82437cc-d2c2-4bac-95c6-1490f5c32c46" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 18:18:54.868927 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.868796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz"
Sep 11 18:18:54.869298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.869268    2582 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"ca908762f0c80dd48b2d392b9d5c8101ad4aac746a71cf08e93ab437eae207dd"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 18:18:54.869339 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:54.869319    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" podUID="e82437cc-d2c2-4bac-95c6-1490f5c32c46" containerName="service-proxy" containerID="cri-o://ca908762f0c80dd48b2d392b9d5c8101ad4aac746a71cf08e93ab437eae207dd" gracePeriod=30
Sep 11 18:18:55.308395 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:55.308359    2582 generic.go:356] "Generic (PLEG): container finished" podID="e82437cc-d2c2-4bac-95c6-1490f5c32c46" containerID="ca908762f0c80dd48b2d392b9d5c8101ad4aac746a71cf08e93ab437eae207dd" exitCode=2
Sep 11 18:18:55.308767 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:55.308420    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerDied","Data":"ca908762f0c80dd48b2d392b9d5c8101ad4aac746a71cf08e93ab437eae207dd"}
Sep 11 18:18:55.308767 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:18:55.308454    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8665d5996-fcrlz" event={"ID":"e82437cc-d2c2-4bac-95c6-1490f5c32c46","Type":"ContainerStarted","Data":"c90d1561111f6d2d9b3b8e4673684c360c69909f0fa5c421b2e0afc239e17736"}
Sep 11 18:19:14.580590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:14.580560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/init-textfile/0.log"
Sep 11 18:19:14.779778 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:14.779747    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/node-exporter/0.log"
Sep 11 18:19:14.977447 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:14.977408    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/kube-rbac-proxy/0.log"
Sep 11 18:19:22.380411 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:22.380380    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlzz5_cc1f7966-16e2-47a2-98c5-084115142030/serve-healthcheck-canary/0.log"
Sep 11 18:19:42.501795 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:42.501736    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:19:42.504044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:42.504010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/972c3842-6619-4463-802a-7a7ca6e6db54-metrics-certs\") pod \"network-metrics-daemon-2x9w6\" (UID: \"972c3842-6619-4463-802a-7a7ca6e6db54\") " pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:19:42.650698 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:42.650662    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bn27r\""
Sep 11 18:19:42.658733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:42.658713    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-2x9w6"
Sep 11 18:19:42.932265 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:42.932240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-2x9w6"]
Sep 11 18:19:42.934788 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:19:42.934760    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972c3842_6619_4463_802a_7a7ca6e6db54.slice/crio-01a6856dbb40e27b678aa7aa2d4e3a8a66a66628c45e9f4860404f5c16145223 WatchSource:0}: Error finding container 01a6856dbb40e27b678aa7aa2d4e3a8a66a66628c45e9f4860404f5c16145223: Status 404 returned error can't find the container with id 01a6856dbb40e27b678aa7aa2d4e3a8a66a66628c45e9f4860404f5c16145223
Sep 11 18:19:43.462621 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:43.462573    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2x9w6" event={"ID":"972c3842-6619-4463-802a-7a7ca6e6db54","Type":"ContainerStarted","Data":"01a6856dbb40e27b678aa7aa2d4e3a8a66a66628c45e9f4860404f5c16145223"}
Sep 11 18:19:44.468235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:44.468196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2x9w6" event={"ID":"972c3842-6619-4463-802a-7a7ca6e6db54","Type":"ContainerStarted","Data":"e721b0625f1b1f7405d2d536618e44b63768cf37414ac585f4f0a5d4a7e4867e"}
Sep 11 18:19:44.468601 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:44.468250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-2x9w6" event={"ID":"972c3842-6619-4463-802a-7a7ca6e6db54","Type":"ContainerStarted","Data":"b5a2a86a3da85301a98cc6e30d98cf2ce2e6ce0b4a1c752bceafb74f2d31d902"}
Sep 11 18:19:44.485089 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:19:44.484988    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-2x9w6" podStartSLOduration=253.695336473 podStartE2EDuration="4m14.484967285s" totalImagesPullingTime="789.630812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:19:44.484115036 +0000 UTC m=+254.370087516" watchObservedRunningTime="2026-09-11 18:19:44.484967285 +0000 UTC m=+254.370939777"
Sep 11 18:20:11.147054 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:20:11.147014    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-98h96" podUID="02b8c5e6-beb4-4aca-a918-284fd81f9210"
Sep 11 18:20:11.553244 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:11.553216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:14.619055 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:14.619017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:14.621326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:14.621307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/02b8c5e6-beb4-4aca-a918-284fd81f9210-metrics-tls\") pod \"dns-default-98h96\" (UID: \"02b8c5e6-beb4-4aca-a918-284fd81f9210\") " pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:14.856135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:14.856100    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bf6d6\""
Sep 11 18:20:14.864244 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:14.864218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:15.142148 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:15.142079    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-98h96"]
Sep 11 18:20:15.144714 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:20:15.144690    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b8c5e6_beb4_4aca_a918_284fd81f9210.slice/crio-29c45d7396cf6f3ae81fa443399e4944988cca3c6d18b61e685c6882aea3d470 WatchSource:0}: Error finding container 29c45d7396cf6f3ae81fa443399e4944988cca3c6d18b61e685c6882aea3d470: Status 404 returned error can't find the container with id 29c45d7396cf6f3ae81fa443399e4944988cca3c6d18b61e685c6882aea3d470
Sep 11 18:20:15.566261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:15.566228    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98h96" event={"ID":"02b8c5e6-beb4-4aca-a918-284fd81f9210","Type":"ContainerStarted","Data":"29c45d7396cf6f3ae81fa443399e4944988cca3c6d18b61e685c6882aea3d470"}
Sep 11 18:20:17.574988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:17.574952    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98h96" event={"ID":"02b8c5e6-beb4-4aca-a918-284fd81f9210","Type":"ContainerStarted","Data":"565b1300d8885aecaf530b74fddf72cffc10b915419b26c7ac73a2657149a1e8"}
Sep 11 18:20:17.575393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:17.575007    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-98h96" event={"ID":"02b8c5e6-beb4-4aca-a918-284fd81f9210","Type":"ContainerStarted","Data":"c4d8221aab9190feb0c88a7959bc366310f32613b5499376435a11c845193ef2"}
Sep 11 18:20:17.575393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:17.575078    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:27.580560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:27.580519    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-98h96"
Sep 11 18:20:27.599142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:27.599085    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-98h96" podStartSLOduration=261.952640358 podStartE2EDuration="4m23.599063623s" totalImagesPullingTime="1.646423265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:16:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:20:17.599900642 +0000 UTC m=+287.485873130" watchObservedRunningTime="2026-09-11 18:20:27.599063623 +0000 UTC m=+297.485036116"
Sep 11 18:20:30.492622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:20:30.492594    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 18:21:54.642422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.642344    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"]
Sep 11 18:21:54.645230 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.645213    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.647507 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.647474    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 18:21:54.647630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.647508    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-g46fw\""
Sep 11 18:21:54.648269 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.648257    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 18:21:54.655670 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.655649    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"]
Sep 11 18:21:54.704593 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.704555    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.704762 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.704606    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wfk7\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-kube-api-access-4wfk7\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.805732 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.805694    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wfk7\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-kube-api-access-4wfk7\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.805911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.805765    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.814020 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.813994    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.814130 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.814086    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wfk7\" (UniqueName: \"kubernetes.io/projected/ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea-kube-api-access-4wfk7\") pod \"cert-manager-cainjector-559cdd8c87-zg9fr\" (UID: \"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:54.953222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:54.953190    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"
Sep 11 18:21:55.219744 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:55.219666    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr"]
Sep 11 18:21:55.222330 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:21:55.222300    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba8a306c_eede_4d82_9fa6_1bdd0b2cd9ea.slice/crio-8b99aea8040809dc3e50fb73b6f5bdb35730b5a93c1ce3ebc97be46d72096811 WatchSource:0}: Error finding container 8b99aea8040809dc3e50fb73b6f5bdb35730b5a93c1ce3ebc97be46d72096811: Status 404 returned error can't find the container with id 8b99aea8040809dc3e50fb73b6f5bdb35730b5a93c1ce3ebc97be46d72096811
Sep 11 18:21:55.224391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:55.224376    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:21:55.894875 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:55.894833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr" event={"ID":"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea","Type":"ContainerStarted","Data":"8b99aea8040809dc3e50fb73b6f5bdb35730b5a93c1ce3ebc97be46d72096811"}
Sep 11 18:21:58.907437 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:58.907394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr" event={"ID":"ba8a306c-eede-4d82-9fa6-1bdd0b2cd9ea","Type":"ContainerStarted","Data":"bc0477d299054834956836ff541c1d416a1ea2cbc707336f97dd8aaa253dee83"}
Sep 11 18:21:58.922518 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:21:58.922466    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-zg9fr" podStartSLOduration=2.061950151 podStartE2EDuration="4.922449406s" totalImagesPullingTime="2.860499255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:21:58.922028355 +0000 UTC m=+388.808000844" watchObservedRunningTime="2026-09-11 18:21:58.922449406 +0000 UTC m=+388.808421894"
Sep 11 18:22:03.235112 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.235077    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5v5n2"]
Sep 11 18:22:03.238020 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.238004    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.240142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.240117    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7w5c2\""
Sep 11 18:22:03.246038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.246012    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5v5n2"]
Sep 11 18:22:03.266285 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.266255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.266447 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.266306    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqzj\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-kube-api-access-9mqzj\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.367591 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.367547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.367591 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.367598    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqzj\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-kube-api-access-9mqzj\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.375184 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.375141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-bound-sa-token\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.375292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.375238    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqzj\" (UniqueName: \"kubernetes.io/projected/d924e1a0-49da-43f4-8f50-8a3aa3373cfa-kube-api-access-9mqzj\") pod \"cert-manager-55b4d954d5-5v5n2\" (UID: \"d924e1a0-49da-43f4-8f50-8a3aa3373cfa\") " pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.546860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.546774    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5v5n2"
Sep 11 18:22:03.814253 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.814146    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5v5n2"]
Sep 11 18:22:03.817783 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:22:03.817755    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd924e1a0_49da_43f4_8f50_8a3aa3373cfa.slice/crio-9f0c5138e2b00a9a0bb085deb572bbe8e770edba9002861a0ca2b1a69bfad89b WatchSource:0}: Error finding container 9f0c5138e2b00a9a0bb085deb572bbe8e770edba9002861a0ca2b1a69bfad89b: Status 404 returned error can't find the container with id 9f0c5138e2b00a9a0bb085deb572bbe8e770edba9002861a0ca2b1a69bfad89b
Sep 11 18:22:03.925749 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.925717    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5v5n2" event={"ID":"d924e1a0-49da-43f4-8f50-8a3aa3373cfa","Type":"ContainerStarted","Data":"2c367a3e3a2990ded3cb8ff25adba09d697bd10c909097b775b50042bb6610ce"}
Sep 11 18:22:03.925893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.925758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5v5n2" event={"ID":"d924e1a0-49da-43f4-8f50-8a3aa3373cfa","Type":"ContainerStarted","Data":"9f0c5138e2b00a9a0bb085deb572bbe8e770edba9002861a0ca2b1a69bfad89b"}
Sep 11 18:22:03.939872 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:03.939828    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5v5n2" podStartSLOduration=0.939813379 podStartE2EDuration="939.813379ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:03.939029483 +0000 UTC m=+393.825001972" watchObservedRunningTime="2026-09-11 18:22:03.939813379 +0000 UTC m=+393.825785868"
Sep 11 18:22:25.566833 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.566797    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"]
Sep 11 18:22:25.569706 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.569689    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.572075 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.572048    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 18:22:25.572221 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.572136    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 18:22:25.572221 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.572199    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 18:22:25.572425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.572410    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8779c\""
Sep 11 18:22:25.572467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.572435    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 18:22:25.587387 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.587362    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"]
Sep 11 18:22:25.628495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.628456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.628495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.628500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.628821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.628566    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.628821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.628602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pgv\" (UniqueName: \"kubernetes.io/projected/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-kube-api-access-67pgv\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.628821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.628628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729259    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pgv\" (UniqueName: \"kubernetes.io/projected/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-kube-api-access-67pgv\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729329    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729391    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.729752 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.729724    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-manifests\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.731691 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.731663    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-tmp\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.731811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.731788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-webhook-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.731876 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.731863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.741346 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.741323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pgv\" (UniqueName: \"kubernetes.io/projected/e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4-kube-api-access-67pgv\") pod \"opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd\" (UID: \"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4\") " pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:25.879768 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:25.879665    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:26.160459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:26.160436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"]
Sep 11 18:22:26.162512 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:22:26.162490    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa2b12_69bc_40f2_b2a8_c11bf076dbc4.slice/crio-9519a49a37b799d8345f3c350bae589ed2d1f0f9baa66416615c519aa65b13a3 WatchSource:0}: Error finding container 9519a49a37b799d8345f3c350bae589ed2d1f0f9baa66416615c519aa65b13a3: Status 404 returned error can't find the container with id 9519a49a37b799d8345f3c350bae589ed2d1f0f9baa66416615c519aa65b13a3
Sep 11 18:22:27.002962 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:27.002916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd" event={"ID":"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4","Type":"ContainerStarted","Data":"9519a49a37b799d8345f3c350bae589ed2d1f0f9baa66416615c519aa65b13a3"}
Sep 11 18:22:30.015575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:30.015538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd" event={"ID":"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4","Type":"ContainerStarted","Data":"89b7e4ac7a8454a21e800bc0196c68ee1a5cfd4313378d822dcf6bac8337e411"}
Sep 11 18:22:31.020432 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:31.020393    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4" containerID="89b7e4ac7a8454a21e800bc0196c68ee1a5cfd4313378d822dcf6bac8337e411" exitCode=0
Sep 11 18:22:31.020825 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:31.020462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd" event={"ID":"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4","Type":"ContainerDied","Data":"89b7e4ac7a8454a21e800bc0196c68ee1a5cfd4313378d822dcf6bac8337e411"}
Sep 11 18:22:32.025996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:32.025910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd" event={"ID":"e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4","Type":"ContainerStarted","Data":"90f119853a70143a61103185ef4aa6eacd652a18ea8f98e7c37a084d68ad9b54"}
Sep 11 18:22:32.026416 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:32.026018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:32.047015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:32.046960    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd" podStartSLOduration=2.302774521 podStartE2EDuration="7.046942132s" totalImagesPullingTime="3.656963401s" totalInitContainerRuntime="1.08720421s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:32.045336061 +0000 UTC m=+421.931308550" watchObservedRunningTime="2026-09-11 18:22:32.046942132 +0000 UTC m=+421.932914622"
Sep 11 18:22:43.031548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:43.031517    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd"
Sep 11 18:22:45.783385 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.783345    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"]
Sep 11 18:22:45.791536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.791509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.794936 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.794918    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 18:22:45.796129 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.796109    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 18:22:45.796129 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.796121    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 18:22:45.796290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.796115    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 18:22:45.813439 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.813414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"]
Sep 11 18:22:45.881510 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.881469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.881510 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.881517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tmp\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.881720 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.881535    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7kd\" (UniqueName: \"kubernetes.io/projected/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-kube-api-access-pv7kd\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.982011 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.981972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.982011 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.982017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tmp\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.982277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.982036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7kd\" (UniqueName: \"kubernetes.io/projected/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-kube-api-access-pv7kd\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.984423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.984385    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tmp\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.984639 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.984618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-tls-certs\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:45.997891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:45.997869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7kd\" (UniqueName: \"kubernetes.io/projected/d02f1a7a-bec9-449e-bf7a-b9e94a5360b7-kube-api-access-pv7kd\") pod \"kube-auth-proxy-78f85fb5d-8dc6w\" (UID: \"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7\") " pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:46.100672 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:46.100591    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"
Sep 11 18:22:46.371783 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:46.371712    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w"]
Sep 11 18:22:46.374775 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:22:46.374751    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02f1a7a_bec9_449e_bf7a_b9e94a5360b7.slice/crio-12e71644e92a805fdbb3991967b384d752cc2c0d1830de62c12b4526d3cac2ba WatchSource:0}: Error finding container 12e71644e92a805fdbb3991967b384d752cc2c0d1830de62c12b4526d3cac2ba: Status 404 returned error can't find the container with id 12e71644e92a805fdbb3991967b384d752cc2c0d1830de62c12b4526d3cac2ba
Sep 11 18:22:47.062233 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.062101    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:22:47.065545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.065518    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.069114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.068630    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 18:22:47.069114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.068873    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lrxfn\""
Sep 11 18:22:47.069114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.068913    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 18:22:47.069114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.068875    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 18:22:47.069416 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.069265    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 18:22:47.080872 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.080844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w" event={"ID":"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7","Type":"ContainerStarted","Data":"12e71644e92a805fdbb3991967b384d752cc2c0d1830de62c12b4526d3cac2ba"}
Sep 11 18:22:47.084443 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.084423    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:22:47.192337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192295    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192411    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94x5\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192741 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192556    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.192741 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.192609    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94x5\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.293814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.293792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.294406 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.294381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.295950 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.295921    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.296044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.295961    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.296315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.296294    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.296397 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.296352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.303932 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.303887    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94x5\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.304560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.304542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-9bzc7\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.377593 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.377510    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:47.665444 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:47.665331    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:22:48.830488 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:22:48.830454    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a5fc3b_5ba9_41e5_9f50_28e2142c3359.slice/crio-de6ab5ff329b573a3ef6ebd6a243b2ebf28b3638cea99e6a70d28a9d7c2cd8bb WatchSource:0}: Error finding container de6ab5ff329b573a3ef6ebd6a243b2ebf28b3638cea99e6a70d28a9d7c2cd8bb: Status 404 returned error can't find the container with id de6ab5ff329b573a3ef6ebd6a243b2ebf28b3638cea99e6a70d28a9d7c2cd8bb
Sep 11 18:22:49.091027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:49.090926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w" event={"ID":"d02f1a7a-bec9-449e-bf7a-b9e94a5360b7","Type":"ContainerStarted","Data":"7846965fd2e8e49c24034516518be8f6d227b7e222fa06120ceb6abd3297d737"}
Sep 11 18:22:49.092252 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:49.092225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" event={"ID":"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359","Type":"ContainerStarted","Data":"de6ab5ff329b573a3ef6ebd6a243b2ebf28b3638cea99e6a70d28a9d7c2cd8bb"}
Sep 11 18:22:49.105853 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:49.105809    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78f85fb5d-8dc6w" podStartSLOduration=1.600356363 podStartE2EDuration="4.10579501s" totalImagesPullingTime="2.505438647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:49.104811413 +0000 UTC m=+438.990783902" watchObservedRunningTime="2026-09-11 18:22:49.10579501 +0000 UTC m=+438.991767496"
Sep 11 18:22:51.007891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:51.007855    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:22:51.008130 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:51.007930    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:22:51.104735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:51.104698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" event={"ID":"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359","Type":"ContainerStarted","Data":"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"}
Sep 11 18:22:51.104918 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:51.104826    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:22:51.124285 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:51.124235    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" podStartSLOduration=1.94913413 podStartE2EDuration="4.124217546s" totalImagesPullingTime="2.175083416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:22:51.122982591 +0000 UTC m=+441.008955085" watchObservedRunningTime="2026-09-11 18:22:51.124217546 +0000 UTC m=+441.010190034"
Sep 11 18:22:52.110753 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:22:52.110722    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:23:04.450993 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.450956    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"]
Sep 11 18:23:04.457506 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.457483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.465767 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.465740    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"]
Sep 11 18:23:04.627368 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627332    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627368 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627366    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4qmb\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-kube-api-access-x4qmb\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627481    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627502    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627523    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.627569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.627542    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728695    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728723    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728757    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.728949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.728830    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4qmb\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-kube-api-access-x4qmb\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.730107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.730074    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.731552 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.731522    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.735578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.735550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.735790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.735765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.735842 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.735794    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.737302 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.737280    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.737790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.737772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4qmb\" (UniqueName: \"kubernetes.io/projected/dbfa15c8-83d8-4812-a12b-ffb6f84f47c2-kube-api-access-x4qmb\") pod \"istiod-openshift-gateway-5677ccb49b-mpvbg\" (UID: \"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:04.766408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:04.766388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:05.060202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.060162    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"]
Sep 11 18:23:05.061808 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:23:05.061779    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfa15c8_83d8_4812_a12b_ffb6f84f47c2.slice/crio-b30c2cd3b2794590b5fe48dde8b192ee6e23cc8f85e6fdbd43ff07dbe93eac41 WatchSource:0}: Error finding container b30c2cd3b2794590b5fe48dde8b192ee6e23cc8f85e6fdbd43ff07dbe93eac41: Status 404 returned error can't find the container with id b30c2cd3b2794590b5fe48dde8b192ee6e23cc8f85e6fdbd43ff07dbe93eac41
Sep 11 18:23:05.066496 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.066462    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:23:05.066608 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.066542    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:23:05.157493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.157458    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg" event={"ID":"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2","Type":"ContainerStarted","Data":"3bf25c586cf504752f4920049283afc12aea545f5b250d097c40b23f9aae12cd"}
Sep 11 18:23:05.157658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.157506    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg" event={"ID":"dbfa15c8-83d8-4812-a12b-ffb6f84f47c2","Type":"ContainerStarted","Data":"b30c2cd3b2794590b5fe48dde8b192ee6e23cc8f85e6fdbd43ff07dbe93eac41"}
Sep 11 18:23:05.157658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.157593    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:05.180751 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:05.180703    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg" podStartSLOduration=1.180686775 podStartE2EDuration="1.180686775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:05.179701025 +0000 UTC m=+455.065673528" watchObservedRunningTime="2026-09-11 18:23:05.180686775 +0000 UTC m=+455.066659263"
Sep 11 18:23:06.163857 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.163831    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mpvbg"
Sep 11 18:23:06.215043 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.215009    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:23:06.215287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.215239    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" podUID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" containerName="discovery" containerID="cri-o://7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1" gracePeriod=30
Sep 11 18:23:06.447343 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.447321    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:23:06.643891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.643858    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.643891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.643897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.643919    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.643947    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g94x5\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.644008    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.644073    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.644117    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs\") pod \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\" (UID: \"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359\") "
Sep 11 18:23:06.644426 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.644389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:23:06.646603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.646568    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token" (OuterVolumeSpecName: "istio-token") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:23:06.646733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.646683    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs" (OuterVolumeSpecName: "local-certs") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:23:06.646733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.646683    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:23:06.646843 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.646794    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts" (OuterVolumeSpecName: "cacerts") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:23:06.646843 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.646811    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5" (OuterVolumeSpecName: "kube-api-access-g94x5") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "kube-api-access-g94x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:23:06.647044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.647019    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" (UID: "f8a5fc3b-5ba9-41e5-9f50-28e2142c3359"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745537    2582 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-kubeconfig\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745567    2582 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-local-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745576    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-dns-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745585    2582 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-cacerts\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745595    2582 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-csr-ca-configmap\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745605    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94x5\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-kube-api-access-g94x5\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:06.745619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:06.745612    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:23:07.167377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.167338    2582 generic.go:356] "Generic (PLEG): container finished" podID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" containerID="7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1" exitCode=0
Sep 11 18:23:07.167808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.167385    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"
Sep 11 18:23:07.167808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.167417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" event={"ID":"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359","Type":"ContainerDied","Data":"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"}
Sep 11 18:23:07.167808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.167467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7" event={"ID":"f8a5fc3b-5ba9-41e5-9f50-28e2142c3359","Type":"ContainerDied","Data":"de6ab5ff329b573a3ef6ebd6a243b2ebf28b3638cea99e6a70d28a9d7c2cd8bb"}
Sep 11 18:23:07.167808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.167508    2582 scope.go:117] "RemoveContainer" containerID="7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"
Sep 11 18:23:07.176628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.176609    2582 scope.go:117] "RemoveContainer" containerID="7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"
Sep 11 18:23:07.176893 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:23:07.176876    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1\": container with ID starting with 7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1 not found: ID does not exist" containerID="7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"
Sep 11 18:23:07.176945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.176901    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1"} err="failed to get container status \"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1\": rpc error: code = NotFound desc = could not find container \"7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1\": container with ID starting with 7f6178a8dc893fd7c4e0cd08666891599ca0da613d379cf5ff5dfe4745794cc1 not found: ID does not exist"
Sep 11 18:23:07.194612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.194586    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:23:07.197593 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.197571    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"]
Sep 11 18:23:07.197844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.197827    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" containerName="discovery"
Sep 11 18:23:07.197844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.197841    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" containerName="discovery"
Sep 11 18:23:07.197970 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.197897    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" containerName="discovery"
Sep 11 18:23:07.200276 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.200258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.202479 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.202457    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 18:23:07.202590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.202542    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w96ft\""
Sep 11 18:23:07.207400 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.207373    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-9bzc7"]
Sep 11 18:23:07.213873 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.213851    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"]
Sep 11 18:23:07.252625 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.252594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.252811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.252742    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2cf\" (UniqueName: \"kubernetes.io/projected/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-kube-api-access-pz2cf\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.353526 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.353488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2cf\" (UniqueName: \"kubernetes.io/projected/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-kube-api-access-pz2cf\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.353714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.353552    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.355957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.355932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.361507 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.361482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2cf\" (UniqueName: \"kubernetes.io/projected/e58588cd-c8c7-4779-b4cb-ae35dd2f4703-kube-api-access-pz2cf\") pod \"llmisvc-controller-manager-777c6bd7d5-4s97g\" (UID: \"e58588cd-c8c7-4779-b4cb-ae35dd2f4703\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.511236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.511118    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:07.780828 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:07.780687    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"]
Sep 11 18:23:07.783090 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:23:07.783065    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode58588cd_c8c7_4779_b4cb_ae35dd2f4703.slice/crio-6c18ec75a67b73367d8bece26583c0e6b5c4b53d958977e44548edd5d6bd63a8 WatchSource:0}: Error finding container 6c18ec75a67b73367d8bece26583c0e6b5c4b53d958977e44548edd5d6bd63a8: Status 404 returned error can't find the container with id 6c18ec75a67b73367d8bece26583c0e6b5c4b53d958977e44548edd5d6bd63a8
Sep 11 18:23:08.173306 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:08.173277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g" event={"ID":"e58588cd-c8c7-4779-b4cb-ae35dd2f4703","Type":"ContainerStarted","Data":"6c18ec75a67b73367d8bece26583c0e6b5c4b53d958977e44548edd5d6bd63a8"}
Sep 11 18:23:08.554425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:08.554277    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8a5fc3b-5ba9-41e5-9f50-28e2142c3359" path="/var/lib/kubelet/pods/f8a5fc3b-5ba9-41e5-9f50-28e2142c3359/volumes"
Sep 11 18:23:11.188216 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:11.188179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g" event={"ID":"e58588cd-c8c7-4779-b4cb-ae35dd2f4703","Type":"ContainerStarted","Data":"4c7a1971d1573996f416c4f1b4cd3b7bb309b2323ac817a694b8494896850c58"}
Sep 11 18:23:11.188631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:11.188261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:23:11.203430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:11.203381    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g" podStartSLOduration=1.43568599 podStartE2EDuration="4.203365405s" totalImagesPullingTime="2.767679415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:23:11.202458705 +0000 UTC m=+461.088431207" watchObservedRunningTime="2026-09-11 18:23:11.203365405 +0000 UTC m=+461.089337893"
Sep 11 18:23:42.194702 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:23:42.194672    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4s97g"
Sep 11 18:24:17.020896 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.020862    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sdbm5"]
Sep 11 18:24:17.025436 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.025419    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:17.027694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.027670    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fgzrr\""
Sep 11 18:24:17.027826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.027810    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 18:24:17.028508 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.028488    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 18:24:17.039005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.038983    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sdbm5"]
Sep 11 18:24:17.082652 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.082618    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7qt\" (UniqueName: \"kubernetes.io/projected/bc24f7e2-c782-4fdf-961b-edfde6fbdd4c-kube-api-access-hm7qt\") pod \"authorino-operator-76b478bd78-sdbm5\" (UID: \"bc24f7e2-c782-4fdf-961b-edfde6fbdd4c\") " pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:17.184013 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.183979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7qt\" (UniqueName: \"kubernetes.io/projected/bc24f7e2-c782-4fdf-961b-edfde6fbdd4c-kube-api-access-hm7qt\") pod \"authorino-operator-76b478bd78-sdbm5\" (UID: \"bc24f7e2-c782-4fdf-961b-edfde6fbdd4c\") " pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:17.194722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.194691    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7qt\" (UniqueName: \"kubernetes.io/projected/bc24f7e2-c782-4fdf-961b-edfde6fbdd4c-kube-api-access-hm7qt\") pod \"authorino-operator-76b478bd78-sdbm5\" (UID: \"bc24f7e2-c782-4fdf-961b-edfde6fbdd4c\") " pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:17.334435 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.334341    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:17.604084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:17.604012    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-sdbm5"]
Sep 11 18:24:17.607157 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:24:17.607130    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc24f7e2_c782_4fdf_961b_edfde6fbdd4c.slice/crio-c1439c2f7172131c59160360b13d3ce2169682df616c81bccdf56125e4b629f5 WatchSource:0}: Error finding container c1439c2f7172131c59160360b13d3ce2169682df616c81bccdf56125e4b629f5: Status 404 returned error can't find the container with id c1439c2f7172131c59160360b13d3ce2169682df616c81bccdf56125e4b629f5
Sep 11 18:24:18.436410 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:18.436373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5" event={"ID":"bc24f7e2-c782-4fdf-961b-edfde6fbdd4c","Type":"ContainerStarted","Data":"c1439c2f7172131c59160360b13d3ce2169682df616c81bccdf56125e4b629f5"}
Sep 11 18:24:20.445643 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:20.445609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5" event={"ID":"bc24f7e2-c782-4fdf-961b-edfde6fbdd4c","Type":"ContainerStarted","Data":"b034228a5fe6493b8a92e9b370a6525afd07631778ef581e27bcb4a140529795"}
Sep 11 18:24:20.446016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:20.445745    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:20.479016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:20.478964    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5" podStartSLOduration=2.32326064 podStartE2EDuration="4.47894797s" totalImagesPullingTime="2.15568733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:20.47848176 +0000 UTC m=+530.364454307" watchObservedRunningTime="2026-09-11 18:24:20.47894797 +0000 UTC m=+530.364920458"
Sep 11 18:24:31.451942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:31.451858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-sdbm5"
Sep 11 18:24:43.912860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:43.912825    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"]
Sep 11 18:24:43.916066 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:43.916048    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:43.919357 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:43.919336    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xz6wx\""
Sep 11 18:24:43.942096 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:43.942074    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"]
Sep 11 18:24:44.004441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.004409    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2m7\" (UniqueName: \"kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.004598 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.004483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.105128 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.105094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2m7\" (UniqueName: \"kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.105298 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.105154    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.105529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.105512    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.114803 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.114781    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2m7\" (UniqueName: \"kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7\") pod \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.226042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.225940    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:44.507257 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.507225    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"]
Sep 11 18:24:44.510298 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:24:44.510275    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fbb3c6_6749_4627_ad69_2163bb9c7681.slice/crio-7dda4fdd8abdf8b705a5d245e10f04b630075e747639cba022174366ab9bd457 WatchSource:0}: Error finding container 7dda4fdd8abdf8b705a5d245e10f04b630075e747639cba022174366ab9bd457: Status 404 returned error can't find the container with id 7dda4fdd8abdf8b705a5d245e10f04b630075e747639cba022174366ab9bd457
Sep 11 18:24:44.540011 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.539984    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s" event={"ID":"04fbb3c6-6749-4627-ad69-2163bb9c7681","Type":"ContainerStarted","Data":"7dda4fdd8abdf8b705a5d245e10f04b630075e747639cba022174366ab9bd457"}
Sep 11 18:24:44.633766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.633732    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"]
Sep 11 18:24:44.641560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.641533    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"]
Sep 11 18:24:44.668731 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.668699    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:24:44.672942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.672926    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.690511 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.690485    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:24:44.810552 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.810477    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmm5c\" (UniqueName: \"kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.810552 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.810521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.912133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.911617    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.912133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.911775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmm5c\" (UniqueName: \"kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.912133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.911962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.929364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.929331    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmm5c\" (UniqueName: \"kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c\") pod \"kuadrant-operator-controller-manager-5f685c496f-4vw28\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:44.984117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:44.984077    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:45.273619 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:45.273582    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:24:45.277556 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:24:45.277516    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d755457_5e9d_4822_b7f5_c0b2d3c76ab4.slice/crio-9c6432fe11074fb9e5ac6a10fcb45c7d6abb68ec876eaca54ed3c202b268fcc4 WatchSource:0}: Error finding container 9c6432fe11074fb9e5ac6a10fcb45c7d6abb68ec876eaca54ed3c202b268fcc4: Status 404 returned error can't find the container with id 9c6432fe11074fb9e5ac6a10fcb45c7d6abb68ec876eaca54ed3c202b268fcc4
Sep 11 18:24:45.547107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:45.547021    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" event={"ID":"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4","Type":"ContainerStarted","Data":"9c6432fe11074fb9e5ac6a10fcb45c7d6abb68ec876eaca54ed3c202b268fcc4"}
Sep 11 18:24:48.562067 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.562009    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" containerName="manager" containerID="cri-o://ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1" gracePeriod=2
Sep 11 18:24:48.797028 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.797002    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:48.799067 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:24:48.799044    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:48.948237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.948206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt2m7\" (UniqueName: \"kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7\") pod \"04fbb3c6-6749-4627-ad69-2163bb9c7681\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") "
Sep 11 18:24:48.948391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.948262    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume\") pod \"04fbb3c6-6749-4627-ad69-2163bb9c7681\" (UID: \"04fbb3c6-6749-4627-ad69-2163bb9c7681\") "
Sep 11 18:24:48.948569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.948542    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "04fbb3c6-6749-4627-ad69-2163bb9c7681" (UID: "04fbb3c6-6749-4627-ad69-2163bb9c7681"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:24:48.950359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:48.950340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7" (OuterVolumeSpecName: "kube-api-access-tt2m7") pod "04fbb3c6-6749-4627-ad69-2163bb9c7681" (UID: "04fbb3c6-6749-4627-ad69-2163bb9c7681"). InnerVolumeSpecName "kube-api-access-tt2m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:24:49.049009 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.048971    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/04fbb3c6-6749-4627-ad69-2163bb9c7681-extensions-socket-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:24:49.049009 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.049003    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2m7\" (UniqueName: \"kubernetes.io/projected/04fbb3c6-6749-4627-ad69-2163bb9c7681-kube-api-access-tt2m7\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:24:49.566629 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.566593    2582 generic.go:356] "Generic (PLEG): container finished" podID="04fbb3c6-6749-4627-ad69-2163bb9c7681" containerID="ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1" exitCode=2
Sep 11 18:24:49.567051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.566644    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:49.567051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.566692    2582 scope.go:117] "RemoveContainer" containerID="ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1"
Sep 11 18:24:49.568760 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:24:49.568729    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:49.575163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.575145    2582 scope.go:117] "RemoveContainer" containerID="ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1"
Sep 11 18:24:49.575468 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:24:49.575451    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1\": container with ID starting with ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1 not found: ID does not exist" containerID="ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1"
Sep 11 18:24:49.575516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:49.575478    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1"} err="failed to get container status \"ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1\": rpc error: code = NotFound desc = could not find container \"ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1\": container with ID starting with ca716ae36cabb96a9e5331914c25e4168e765ef7d6dcb8aed69ae0bcb82e87b1 not found: ID does not exist"
Sep 11 18:24:49.576867 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:24:49.576849    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:50.551537 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:24:50.551498    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5f685c496f-hvg7s\" is forbidden: User \"system:node:ip-10-0-128-148.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-148.ec2.internal' and this object" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-hvg7s"
Sep 11 18:24:50.555287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:50.555244    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" path="/var/lib/kubelet/pods/04fbb3c6-6749-4627-ad69-2163bb9c7681/volumes"
Sep 11 18:24:56.598480 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:56.598401    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" event={"ID":"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4","Type":"ContainerStarted","Data":"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"}
Sep 11 18:24:56.598480 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:56.598463    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:24:56.626996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:24:56.626947    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" podStartSLOduration=1.675189616 podStartE2EDuration="12.626929709s" totalImagesPullingTime="10.951740093s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:24:56.625983796 +0000 UTC m=+566.511956286" watchObservedRunningTime="2026-09-11 18:24:56.626929709 +0000 UTC m=+566.512902197"
Sep 11 18:25:07.605042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.605013    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:25:07.914693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.914661    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:25:07.914961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.914949    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" containerName="manager"
Sep 11 18:25:07.915002 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.914963    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" containerName="manager"
Sep 11 18:25:07.915043 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.915024    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="04fbb3c6-6749-4627-ad69-2163bb9c7681" containerName="manager"
Sep 11 18:25:07.917867 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.917852    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:07.940268 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:07.940241    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:25:08.105463 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.105432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.105636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.105494    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7z2t\" (UniqueName: \"kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.206678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.206595    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.206678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.206652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7z2t\" (UniqueName: \"kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.206988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.206968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.215530 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.215509    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7z2t\" (UniqueName: \"kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t\") pod \"kuadrant-operator-controller-manager-5769f99d9f-xr5wf\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.227139 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.227120    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.502023 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.501996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:25:08.504158 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:25:08.504132    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda90cf07_4257_47ea_807b_47629aaec120.slice/crio-48b82300b90dc9678cf7c3a8cbc1b8adcb47f69c0757d90d79fb6b7b7571bfbf WatchSource:0}: Error finding container 48b82300b90dc9678cf7c3a8cbc1b8adcb47f69c0757d90d79fb6b7b7571bfbf: Status 404 returned error can't find the container with id 48b82300b90dc9678cf7c3a8cbc1b8adcb47f69c0757d90d79fb6b7b7571bfbf
Sep 11 18:25:08.651363 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.651314    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerStarted","Data":"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"}
Sep 11 18:25:08.651844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.651382    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerStarted","Data":"48b82300b90dc9678cf7c3a8cbc1b8adcb47f69c0757d90d79fb6b7b7571bfbf"}
Sep 11 18:25:08.651844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.651432    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:08.671896 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:08.671810    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" podStartSLOduration=1.6717896140000001 podStartE2EDuration="1.671789614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:25:08.671433294 +0000 UTC m=+578.557405808" watchObservedRunningTime="2026-09-11 18:25:08.671789614 +0000 UTC m=+578.557762104"
Sep 11 18:25:19.658009 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.657980    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:25:19.709717 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.709682    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:25:19.709904 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.709879    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" podUID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" containerName="manager" containerID="cri-o://5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef" gracePeriod=10
Sep 11 18:25:19.953888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.953866    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:25:19.994004 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.993973    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume\") pod \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") "
Sep 11 18:25:19.994140 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.994013    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmm5c\" (UniqueName: \"kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c\") pod \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\" (UID: \"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4\") "
Sep 11 18:25:19.994457 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.994437    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" (UID: "2d755457-5e9d-4822-b7f5-c0b2d3c76ab4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:25:19.996050 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:19.996030    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c" (OuterVolumeSpecName: "kube-api-access-vmm5c") pod "2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" (UID: "2d755457-5e9d-4822-b7f5-c0b2d3c76ab4"). InnerVolumeSpecName "kube-api-access-vmm5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:25:20.094742 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.094704    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-extensions-socket-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:25:20.094742 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.094738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmm5c\" (UniqueName: \"kubernetes.io/projected/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4-kube-api-access-vmm5c\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:25:20.702560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.702527    2582 generic.go:356] "Generic (PLEG): container finished" podID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" containerID="5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef" exitCode=0
Sep 11 18:25:20.703047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.702575    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"
Sep 11 18:25:20.703047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.702607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" event={"ID":"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4","Type":"ContainerDied","Data":"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"}
Sep 11 18:25:20.703047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.702656    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28" event={"ID":"2d755457-5e9d-4822-b7f5-c0b2d3c76ab4","Type":"ContainerDied","Data":"9c6432fe11074fb9e5ac6a10fcb45c7d6abb68ec876eaca54ed3c202b268fcc4"}
Sep 11 18:25:20.703047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.702677    2582 scope.go:117] "RemoveContainer" containerID="5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"
Sep 11 18:25:20.711388 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.711139    2582 scope.go:117] "RemoveContainer" containerID="5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"
Sep 11 18:25:20.711475 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:25:20.711457    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef\": container with ID starting with 5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef not found: ID does not exist" containerID="5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"
Sep 11 18:25:20.711511 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.711483    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef"} err="failed to get container status \"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef\": rpc error: code = NotFound desc = could not find container \"5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef\": container with ID starting with 5e83defc94ef20b1a0e660f88a5885bc265a0104ae810f8b627f92d4c2e7c5ef not found: ID does not exist"
Sep 11 18:25:20.725489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.725462    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:25:20.729913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:20.729891    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5f685c496f-4vw28"]
Sep 11 18:25:22.553051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:25:22.553020    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" path="/var/lib/kubelet/pods/2d755457-5e9d-4822-b7f5-c0b2d3c76ab4/volumes"
Sep 11 18:26:45.025627 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.025588    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"]
Sep 11 18:26:45.027915 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.025875    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" containerName="manager"
Sep 11 18:26:45.027915 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.025886    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" containerName="manager"
Sep 11 18:26:45.027915 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.025938    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d755457-5e9d-4822-b7f5-c0b2d3c76ab4" containerName="manager"
Sep 11 18:26:45.028777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.028744    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.031623 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.031602    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 18:26:45.031744 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.031648    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-8zxk6\""
Sep 11 18:26:45.040910 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.040887    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"]
Sep 11 18:26:45.150499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.150463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7daac5-40f4-4aea-b77b-54ac139c013b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.150651 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.150509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ffpl\" (UniqueName: \"kubernetes.io/projected/be7daac5-40f4-4aea-b77b-54ac139c013b-kube-api-access-2ffpl\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.251087 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.251054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7daac5-40f4-4aea-b77b-54ac139c013b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.251290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.251100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ffpl\" (UniqueName: \"kubernetes.io/projected/be7daac5-40f4-4aea-b77b-54ac139c013b-kube-api-access-2ffpl\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.251996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.251972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be7daac5-40f4-4aea-b77b-54ac139c013b-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.258936 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.258911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ffpl\" (UniqueName: \"kubernetes.io/projected/be7daac5-40f4-4aea-b77b-54ac139c013b-kube-api-access-2ffpl\") pod \"payload-pre-processing-59fbc47bf8-lcqqd\" (UID: \"be7daac5-40f4-4aea-b77b-54ac139c013b\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.339082 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.339001    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:45.608964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:45.608937    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"]
Sep 11 18:26:45.611657 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:26:45.611614    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe7daac5_40f4_4aea_b77b_54ac139c013b.slice/crio-e74c5137b4de158bab341ff971bfddd18fdd5a7fa2ff374c18912e194870b47e WatchSource:0}: Error finding container e74c5137b4de158bab341ff971bfddd18fdd5a7fa2ff374c18912e194870b47e: Status 404 returned error can't find the container with id e74c5137b4de158bab341ff971bfddd18fdd5a7fa2ff374c18912e194870b47e
Sep 11 18:26:46.051473 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:46.051436    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd" event={"ID":"be7daac5-40f4-4aea-b77b-54ac139c013b","Type":"ContainerStarted","Data":"e74c5137b4de158bab341ff971bfddd18fdd5a7fa2ff374c18912e194870b47e"}
Sep 11 18:26:47.056725 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:47.056686    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd" event={"ID":"be7daac5-40f4-4aea-b77b-54ac139c013b","Type":"ContainerStarted","Data":"7f1bd07426dceeaa92e00b70b278757a878089bbddabbb0952b35764bc435cda"}
Sep 11 18:26:47.057136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:47.056809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:26:47.074943 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:47.074896    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd" podStartSLOduration=0.762178723 podStartE2EDuration="2.074880556s" totalImagesPullingTime="1.312701833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:26:47.074123876 +0000 UTC m=+676.960096366" watchObservedRunningTime="2026-09-11 18:26:47.074880556 +0000 UTC m=+676.960853228"
Sep 11 18:26:58.064136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:26:58.064100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-lcqqd"
Sep 11 18:27:10.048882 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.048847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"]
Sep 11 18:27:10.051949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.051927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.055313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.055285    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 11 18:27:10.055313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.055306    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 18:27:10.055486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.055286    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6klqx\""
Sep 11 18:27:10.055486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.055294    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 18:27:10.060432 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.060409    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"]
Sep 11 18:27:10.151189 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.151359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151206    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4w8\" (UniqueName: \"kubernetes.io/projected/96e8ae04-711f-4acb-84a9-a241e774aec5-kube-api-access-fq4w8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.151359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.151359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151278    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8ae04-711f-4acb-84a9-a241e774aec5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.151359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151293    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.151359 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.151309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.251828 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251789    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.251828 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251832    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4w8\" (UniqueName: \"kubernetes.io/projected/96e8ae04-711f-4acb-84a9-a241e774aec5-kube-api-access-fq4w8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251870    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8ae04-711f-4acb-84a9-a241e774aec5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251900    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.251914    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252328 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.252300    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252399 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.252337    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.252399 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.252390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.254238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.254219    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e8ae04-711f-4acb-84a9-a241e774aec5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.254443 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.254427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e8ae04-711f-4acb-84a9-a241e774aec5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.261935 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.261914    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4w8\" (UniqueName: \"kubernetes.io/projected/96e8ae04-711f-4acb-84a9-a241e774aec5-kube-api-access-fq4w8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7\" (UID: \"96e8ae04-711f-4acb-84a9-a241e774aec5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.363186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.363093    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:10.635772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.635640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"]
Sep 11 18:27:10.638440 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:27:10.638409    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e8ae04_711f_4acb_84a9_a241e774aec5.slice/crio-83e6b8a1897c042f07040ef5c52a668d6768c646431bb48731b7235da62c986b WatchSource:0}: Error finding container 83e6b8a1897c042f07040ef5c52a668d6768c646431bb48731b7235da62c986b: Status 404 returned error can't find the container with id 83e6b8a1897c042f07040ef5c52a668d6768c646431bb48731b7235da62c986b
Sep 11 18:27:10.639940 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:10.639921    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:27:11.159794 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:11.159748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7" event={"ID":"96e8ae04-711f-4acb-84a9-a241e774aec5","Type":"ContainerStarted","Data":"83e6b8a1897c042f07040ef5c52a668d6768c646431bb48731b7235da62c986b"}
Sep 11 18:27:12.165878 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:12.165841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7" event={"ID":"96e8ae04-711f-4acb-84a9-a241e774aec5","Type":"ContainerStarted","Data":"dd1db8271ba88e4c0407580a12fb23002653aaf6fb1e636285ad6be8254dacde"}
Sep 11 18:27:12.181182 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:12.181134    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7" podStartSLOduration=1.1886119800000001 podStartE2EDuration="2.181119683s" totalImagesPullingTime="992.507703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:12.180636169 +0000 UTC m=+702.066608669" watchObservedRunningTime="2026-09-11 18:27:12.181119683 +0000 UTC m=+702.067092171"
Sep 11 18:27:20.364266 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:20.364230    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:20.364745 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:20.364330    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:20.376807 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:20.376779    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:21.212774 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:21.212747    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7"
Sep 11 18:27:45.131405 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.131325    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"]
Sep 11 18:27:45.135677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.135654    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.138074 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.138045    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 11 18:27:45.144633 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.144608    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"]
Sep 11 18:27:45.213116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.213314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213121    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.213314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213147    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.213314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.213314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.213314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.213297    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfgs2\" (UniqueName: \"kubernetes.io/projected/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-kube-api-access-gfgs2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314205    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314238 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314242    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314266    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314286    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314313    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfgs2\" (UniqueName: \"kubernetes.io/projected/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-kube-api-access-gfgs2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314756 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.314806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.314768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.316614 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.316590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.316746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.316731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.322063 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.322043    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfgs2\" (UniqueName: \"kubernetes.io/projected/bc958ea4-57f3-4561-9d94-bc4821f8fb3a-kube-api-access-gfgs2\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7\" (UID: \"bc958ea4-57f3-4561-9d94-bc4821f8fb3a\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.447819 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.447778    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:45.725529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:45.724011    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"]
Sep 11 18:27:45.729379 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:27:45.729348    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc958ea4_57f3_4561_9d94_bc4821f8fb3a.slice/crio-b72d182765b09185f5797c34119db6180556ebb808aaacec044497755f0ea61e WatchSource:0}: Error finding container b72d182765b09185f5797c34119db6180556ebb808aaacec044497755f0ea61e: Status 404 returned error can't find the container with id b72d182765b09185f5797c34119db6180556ebb808aaacec044497755f0ea61e
Sep 11 18:27:46.312525 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:46.312483    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7" event={"ID":"bc958ea4-57f3-4561-9d94-bc4821f8fb3a","Type":"ContainerStarted","Data":"7d42249b24da8370975a0ad17958e8dbc6eae185cc74a9a4cfb070ffcb81abbe"}
Sep 11 18:27:46.312897 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:46.312540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7" event={"ID":"bc958ea4-57f3-4561-9d94-bc4821f8fb3a","Type":"ContainerStarted","Data":"b72d182765b09185f5797c34119db6180556ebb808aaacec044497755f0ea61e"}
Sep 11 18:27:46.331515 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:46.331461    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7" podStartSLOduration=1.144800595 podStartE2EDuration="1.331444643s" totalImagesPullingTime="186.644048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:27:46.329726409 +0000 UTC m=+736.215698899" watchObservedRunningTime="2026-09-11 18:27:46.331444643 +0000 UTC m=+736.217417132"
Sep 11 18:27:55.448382 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:55.448341    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:55.448852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:55.448408    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:55.460897 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:55.460873    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:27:56.361964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:27:56.361929    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7"
Sep 11 18:29:51.703358 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.703320    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:29:51.706502 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.706479    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.709481 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.709461    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-52b585-openshift-default-dockercfg-v242w\""
Sep 11 18:29:51.726286 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.726265    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:29:51.745649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745622    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745761 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745761 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745798    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745821    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745959 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gld4g\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745959 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.745959 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.745947    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.764663 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.764631    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:29:51.767995 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.767975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.770315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.770293    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c8b274-openshift-default-dockercfg-hbbww\""
Sep 11 18:29:51.782615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.782585    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:29:51.847056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847067    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847187    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847353    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gld4g\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zknvr\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847625    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847691    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.847938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.847938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.848147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.847972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.848147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.848071    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.848266 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.848157    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.848324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.848288    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.849925 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.849898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.850443 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.850419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.862707 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.862685    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.867851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.867825    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gld4g\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g\") pod \"e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") " pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:51.948305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948269    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948310    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948340    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zknvr\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948486    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948861 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948846    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.948946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.948928    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.949027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.949006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.949075 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.949044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.949133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.949111    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.950851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.950815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.951102 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.951085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.960891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.960833    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:51.961461 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:51.961440    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zknvr\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr\") pod \"e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") " pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:52.015991 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.015957    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:52.077930 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.077745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:52.296902 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.296877    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:29:52.298913 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:29:52.298888    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7481c80_a626_4b7d_a4f5_be4848f80c85.slice/crio-8c48817ea417040fb228c15aadda22a7109d0b9ae063f9054e293778bc113ee0 WatchSource:0}: Error finding container 8c48817ea417040fb228c15aadda22a7109d0b9ae063f9054e293778bc113ee0: Status 404 returned error can't find the container with id 8c48817ea417040fb228c15aadda22a7109d0b9ae063f9054e293778bc113ee0
Sep 11 18:29:52.362700 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.362676    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:29:52.364882 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:29:52.364855    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86aa2cca_2673_488d_83f3_f198be21d6bb.slice/crio-7b1ba1412bbb5919884c0683cfdd780f1eb59964a3ec545d2d9ab4cdd35c3026 WatchSource:0}: Error finding container 7b1ba1412bbb5919884c0683cfdd780f1eb59964a3ec545d2d9ab4cdd35c3026: Status 404 returned error can't find the container with id 7b1ba1412bbb5919884c0683cfdd780f1eb59964a3ec545d2d9ab4cdd35c3026
Sep 11 18:29:52.858583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.858551    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerStarted","Data":"7b1ba1412bbb5919884c0683cfdd780f1eb59964a3ec545d2d9ab4cdd35c3026"}
Sep 11 18:29:52.859763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:52.859732    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerStarted","Data":"8c48817ea417040fb228c15aadda22a7109d0b9ae063f9054e293778bc113ee0"}
Sep 11 18:29:55.258895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.258848    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.259275 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.258944    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.259275 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.258988    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.264913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.264864    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.265040 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.264959    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.265040 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.265011    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:29:55.875395 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.875353    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerStarted","Data":"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"}
Sep 11 18:29:55.876826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.876801    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerStarted","Data":"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"}
Sep 11 18:29:55.993191 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:55.993122    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podStartSLOduration=2.101619201 podStartE2EDuration="4.99310474s" totalImagesPullingTime="2.891485539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:55.929724407 +0000 UTC m=+865.815696897" watchObservedRunningTime="2026-09-11 18:29:55.99310474 +0000 UTC m=+865.879077229"
Sep 11 18:29:56.016303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.016267    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:56.021185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.021145    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:56.063682 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.063630    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podStartSLOduration=2.100069755 podStartE2EDuration="5.063613316s" totalImagesPullingTime="2.963543561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:55.993252121 +0000 UTC m=+865.879224611" watchObservedRunningTime="2026-09-11 18:29:56.063613316 +0000 UTC m=+865.949585805"
Sep 11 18:29:56.078408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.078374    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:56.083341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.083316    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:56.753806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.753776    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:29:56.757073 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.757050    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.760907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.760879    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:29:56.760907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.760887    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lvtx6\""
Sep 11 18:29:56.761163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.760973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-b63ece-serving-cert\""
Sep 11 18:29:56.761163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.760990    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:29:56.761163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.761130    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:29:56.774163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.774133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:29:56.881811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.881777    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:56.882000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.881834    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:56.882586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.882561    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:29:56.882776 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.882757    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:29:56.894941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.894913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.895068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.894953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr54m\" (UniqueName: \"kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.895118 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.895068    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.895152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.895140    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.996520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.996480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.996693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.996541    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr54m\" (UniqueName: \"kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.996693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.996628    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.996860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.996841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.997471 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:29:56.997445    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-serving-cert: secret "maas-api-e2e-worker-w0-b63ece-serving-cert" not found
Sep 11 18:29:56.997603 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:29:56.997531    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls podName:bdc80574-906b-4040-8b7b-8ff574470fd9 nodeName:}" failed. No retries permitted until 2026-09-11 18:29:57.497510819 +0000 UTC m=+867.383483288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls") pod "maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" (UID: "bdc80574-906b-4040-8b7b-8ff574470fd9") : secret "maas-api-e2e-worker-w0-b63ece-serving-cert" not found
Sep 11 18:29:56.997678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.997596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:56.999195 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:56.999153    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:57.006995 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.006939    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr54m\" (UniqueName: \"kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:57.501543 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.501507    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:57.503864 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.503837    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") pod \"maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:57.666862 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.666825    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:29:57.770058 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.769978    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:29:57.773379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.773360    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:57.776697 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.776670    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-b63ece\""
Sep 11 18:29:57.776820 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.776764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-b63ece-dockercfg-744xf\""
Sep 11 18:29:57.790021 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.789992    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:29:57.859422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.859390    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:29:57.863117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.863087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:57.880519 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.880495    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:29:57.904281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.904249    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67pc4\" (UniqueName: \"kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:57.904436 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.904294    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:57.948281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:57.948257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:29:57.949548 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:29:57.949523    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc80574_906b_4040_8b7b_8ff574470fd9.slice/crio-e83b4487b5e3a1ea465f50fd2270f76ad0de63d1044081c3afaf2964654b79ad WatchSource:0}: Error finding container e83b4487b5e3a1ea465f50fd2270f76ad0de63d1044081c3afaf2964654b79ad: Status 404 returned error can't find the container with id e83b4487b5e3a1ea465f50fd2270f76ad0de63d1044081c3afaf2964654b79ad
Sep 11 18:29:58.005652 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.005620    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67pc4\" (UniqueName: \"kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.005800 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.005681    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.005800 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.005716    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmsxc\" (UniqueName: \"kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.005990 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.005966    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.006520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.006501    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.016847 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.016821    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67pc4\" (UniqueName: \"kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4\") pod \"payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.083807 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.083708    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.107208 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.107145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmsxc\" (UniqueName: \"kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.107353 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.107251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.108104 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.108081    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.116913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.116886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmsxc\" (UniqueName: \"kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc\") pod \"payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.173724 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.173683    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.363860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.363774    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:29:58.367165 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:29:58.367138    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4fe3f3_bdc9_4e59_88c7_65b6a2e57258.slice/crio-e4c664910f13716c06fb2f029d7b31e2f20bec5b19f12bd7fc286ff0918c06ed WatchSource:0}: Error finding container e4c664910f13716c06fb2f029d7b31e2f20bec5b19f12bd7fc286ff0918c06ed: Status 404 returned error can't find the container with id e4c664910f13716c06fb2f029d7b31e2f20bec5b19f12bd7fc286ff0918c06ed
Sep 11 18:29:58.474193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.474148    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:29:58.476034 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:29:58.476007    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b6520a_3126_4282_abd9_839c7d653723.slice/crio-08f712d301a80e758288c7e69d3e1156f9c34f633d5d35ca91bc9b54fdc8783d WatchSource:0}: Error finding container 08f712d301a80e758288c7e69d3e1156f9c34f633d5d35ca91bc9b54fdc8783d: Status 404 returned error can't find the container with id 08f712d301a80e758288c7e69d3e1156f9c34f633d5d35ca91bc9b54fdc8783d
Sep 11 18:29:58.897859 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.897372    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" event={"ID":"28b6520a-3126-4282-abd9-839c7d653723","Type":"ContainerStarted","Data":"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"}
Sep 11 18:29:58.897859 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.897440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" event={"ID":"28b6520a-3126-4282-abd9-839c7d653723","Type":"ContainerStarted","Data":"08f712d301a80e758288c7e69d3e1156f9c34f633d5d35ca91bc9b54fdc8783d"}
Sep 11 18:29:58.897859 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.897598    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:29:58.899484 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.899453    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" event={"ID":"bdc80574-906b-4040-8b7b-8ff574470fd9","Type":"ContainerStarted","Data":"e83b4487b5e3a1ea465f50fd2270f76ad0de63d1044081c3afaf2964654b79ad"}
Sep 11 18:29:58.903957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.902564    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" event={"ID":"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258","Type":"ContainerStarted","Data":"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"}
Sep 11 18:29:58.903957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.902609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" event={"ID":"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258","Type":"ContainerStarted","Data":"e4c664910f13716c06fb2f029d7b31e2f20bec5b19f12bd7fc286ff0918c06ed"}
Sep 11 18:29:58.903957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.902639    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:29:58.928678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.928619    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" podStartSLOduration=1.928602376 podStartE2EDuration="1.928602376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:58.928364999 +0000 UTC m=+868.814337488" watchObservedRunningTime="2026-09-11 18:29:58.928602376 +0000 UTC m=+868.814574865"
Sep 11 18:29:58.977856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:29:58.977792    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" podStartSLOduration=1.9777698080000001 podStartE2EDuration="1.977769808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:29:58.977389814 +0000 UTC m=+868.863362301" watchObservedRunningTime="2026-09-11 18:29:58.977769808 +0000 UTC m=+868.863742298"
Sep 11 18:30:00.914411 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:00.914373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" event={"ID":"bdc80574-906b-4040-8b7b-8ff574470fd9","Type":"ContainerStarted","Data":"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"}
Sep 11 18:30:00.914799 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:00.914520    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:30:00.940629 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:00.940567    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" podStartSLOduration=2.305585234 podStartE2EDuration="4.940553032s" totalImagesPullingTime="2.634967798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:00.940471395 +0000 UTC m=+870.826443884" watchObservedRunningTime="2026-09-11 18:30:00.940553032 +0000 UTC m=+870.826525521"
Sep 11 18:30:02.355773 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.355737    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:30:02.358837 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.358819    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.361164 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.361142    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-c8b274-dockercfg-tt546\""
Sep 11 18:30:02.361460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.361446    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-c8b274\""
Sep 11 18:30:02.369386 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.369363    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:30:02.451644 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.451599    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tg6j\" (UniqueName: \"kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.451644 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.451658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.552743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.552713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tg6j\" (UniqueName: \"kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.552941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.552762    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.553545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.553521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.561333 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.561310    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tg6j\" (UniqueName: \"kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j\") pod \"payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.593281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.593246    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:30:02.596748 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.596730    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.610026 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.609966    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:30:02.668616 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.668588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:02.754459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.754422    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td4v2\" (UniqueName: \"kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.754596 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.754503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.855858 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.855818    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.856035 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.855979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td4v2\" (UniqueName: \"kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.856644 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.856623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.866559 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.866502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td4v2\" (UniqueName: \"kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2\") pod \"payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.907465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.907434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:02.952314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:02.952284    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:30:02.953988 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:02.953964    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod428aa947_530b_4eb7_84db_b39697612f30.slice/crio-409e8dba7c69e7c25db57edffe7d063d01d88e65b1aa492af80501791a8cc42d WatchSource:0}: Error finding container 409e8dba7c69e7c25db57edffe7d063d01d88e65b1aa492af80501791a8cc42d: Status 404 returned error can't find the container with id 409e8dba7c69e7c25db57edffe7d063d01d88e65b1aa492af80501791a8cc42d
Sep 11 18:30:03.192665 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.192634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:30:03.194013 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:03.193970    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod203b723e_af7f_4949_a05e_53a5a99d7fc3.slice/crio-a53e1eadcbea67c7661eaaf8f0e2d5a8c156ce7fab5ddc20efd6aba375ac4aa8 WatchSource:0}: Error finding container a53e1eadcbea67c7661eaaf8f0e2d5a8c156ce7fab5ddc20efd6aba375ac4aa8: Status 404 returned error can't find the container with id a53e1eadcbea67c7661eaaf8f0e2d5a8c156ce7fab5ddc20efd6aba375ac4aa8
Sep 11 18:30:03.931473 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.931417    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" event={"ID":"428aa947-530b-4eb7-84db-b39697612f30","Type":"ContainerStarted","Data":"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"}
Sep 11 18:30:03.931911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.931609    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:03.931911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.931651    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" event={"ID":"428aa947-530b-4eb7-84db-b39697612f30","Type":"ContainerStarted","Data":"409e8dba7c69e7c25db57edffe7d063d01d88e65b1aa492af80501791a8cc42d"}
Sep 11 18:30:03.933035 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.933012    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" event={"ID":"203b723e-af7f-4949-a05e-53a5a99d7fc3","Type":"ContainerStarted","Data":"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"}
Sep 11 18:30:03.933210 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.933051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" event={"ID":"203b723e-af7f-4949-a05e-53a5a99d7fc3","Type":"ContainerStarted","Data":"a53e1eadcbea67c7661eaaf8f0e2d5a8c156ce7fab5ddc20efd6aba375ac4aa8"}
Sep 11 18:30:03.933210 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.933141    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:03.957244 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.957197    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" podStartSLOduration=1.957180752 podStartE2EDuration="1.957180752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:03.954629349 +0000 UTC m=+873.840601840" watchObservedRunningTime="2026-09-11 18:30:03.957180752 +0000 UTC m=+873.843153231"
Sep 11 18:30:03.981987 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:03.981932    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" podStartSLOduration=1.981914656 podStartE2EDuration="1.981914656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:03.980728414 +0000 UTC m=+873.866700903" watchObservedRunningTime="2026-09-11 18:30:03.981914656 +0000 UTC m=+873.867887191"
Sep 11 18:30:04.287692 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.287605    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:30:04.291666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.291640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.295048 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.295023    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-52b585\""
Sep 11 18:30:04.295203 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.295031    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-52b585-dockercfg-w8xnx\""
Sep 11 18:30:04.305287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.305264    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:30:04.371018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.370984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9lh\" (UniqueName: \"kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.371166 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.371027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.472160 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.472119    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9lh\" (UniqueName: \"kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.472160 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.472165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.472857 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.472835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.481739 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.481715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9lh\" (UniqueName: \"kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh\") pod \"payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") " pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.603382 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.603302    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:04.887775 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.887747    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:30:04.889805 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:04.889780    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fb1f08_8a3a_4ec5_b6e2_009ef20c28d1.slice/crio-1379b59e70e155eb042a96c1a4af00e859604274b1576c664f4cf38e73372dcc WatchSource:0}: Error finding container 1379b59e70e155eb042a96c1a4af00e859604274b1576c664f4cf38e73372dcc: Status 404 returned error can't find the container with id 1379b59e70e155eb042a96c1a4af00e859604274b1576c664f4cf38e73372dcc
Sep 11 18:30:04.940069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:04.940041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" event={"ID":"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1","Type":"ContainerStarted","Data":"1379b59e70e155eb042a96c1a4af00e859604274b1576c664f4cf38e73372dcc"}
Sep 11 18:30:05.946975 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:05.946939    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" event={"ID":"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1","Type":"ContainerStarted","Data":"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"}
Sep 11 18:30:05.947489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:05.947037    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:05.974095 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:05.974040    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" podStartSLOduration=1.974022975 podStartE2EDuration="1.974022975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:05.972466162 +0000 UTC m=+875.858438674" watchObservedRunningTime="2026-09-11 18:30:05.974022975 +0000 UTC m=+875.859995464"
Sep 11 18:30:06.927793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:06.927762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:30:07.024893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.024857    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:30:07.028161 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.028140    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.030483 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.030459    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-71b1c2-dockercfg-xbrm2\""
Sep 11 18:30:07.030832 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.030811    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-71b1c2\""
Sep 11 18:30:07.043558 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.043533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:30:07.205733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.205635    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvrfl\" (UniqueName: \"kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.205891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.205747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.307030 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.306996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvrfl\" (UniqueName: \"kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.307228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.307076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.307846 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.307824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.318592 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.318568    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvrfl\" (UniqueName: \"kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl\") pod \"payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.339536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.339507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.624077 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.624054    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:30:07.626577 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:07.626551    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d71c16_b624_4cee_a347_d33070c7a781.slice/crio-798a62d19cff8bb286f75733b974d14a956b350be03db379ce1743f743f3c767 WatchSource:0}: Error finding container 798a62d19cff8bb286f75733b974d14a956b350be03db379ce1743f743f3c767: Status 404 returned error can't find the container with id 798a62d19cff8bb286f75733b974d14a956b350be03db379ce1743f743f3c767
Sep 11 18:30:07.958516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.958465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" event={"ID":"94d71c16-b624-4cee-a347-d33070c7a781","Type":"ContainerStarted","Data":"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"}
Sep 11 18:30:07.958710 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.958651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:07.958710 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.958678    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" event={"ID":"94d71c16-b624-4cee-a347-d33070c7a781","Type":"ContainerStarted","Data":"798a62d19cff8bb286f75733b974d14a956b350be03db379ce1743f743f3c767"}
Sep 11 18:30:07.995885 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:07.995823    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" podStartSLOduration=0.995806181 podStartE2EDuration="995.806181ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:07.994389704 +0000 UTC m=+877.880362195" watchObservedRunningTime="2026-09-11 18:30:07.995806181 +0000 UTC m=+877.881778666"
Sep 11 18:30:09.910034 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:09.910003    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:30:09.910412 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:09.910111    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:30:11.754694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.754655    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:11.758364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.758345    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.760687 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.760663    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aa3ba758\""
Sep 11 18:30:11.767928 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.767900    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:11.849972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.849942    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.849972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.849975    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwt2\" (UniqueName: \"kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.951536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.951500    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.951536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.951536    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwt2\" (UniqueName: \"kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.952296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.952278    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:11.960492 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:11.960462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwt2\" (UniqueName: \"kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2\") pod \"payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:12.069210 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:12.069079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:12.341671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:12.341645    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:12.343456 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:12.343430    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdc3480_57c9_4419_b185_51546d065e02.slice/crio-61f0e45229f3e3a69b293bc7bf5438b1b550c099f2cdcf73eb5966a5d2119775 WatchSource:0}: Error finding container 61f0e45229f3e3a69b293bc7bf5438b1b550c099f2cdcf73eb5966a5d2119775: Status 404 returned error can't find the container with id 61f0e45229f3e3a69b293bc7bf5438b1b550c099f2cdcf73eb5966a5d2119775
Sep 11 18:30:12.983462 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:12.983427    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" event={"ID":"dbdc3480-57c9-4419-b185-51546d065e02","Type":"ContainerStarted","Data":"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"}
Sep 11 18:30:12.983869 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:12.983478    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" event={"ID":"dbdc3480-57c9-4419-b185-51546d065e02","Type":"ContainerStarted","Data":"61f0e45229f3e3a69b293bc7bf5438b1b550c099f2cdcf73eb5966a5d2119775"}
Sep 11 18:30:12.983869 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:12.983536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:13.000357 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:13.000310    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" podStartSLOduration=2.00029447 podStartE2EDuration="2.00029447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:12.99889077 +0000 UTC m=+882.884863262" watchObservedRunningTime="2026-09-11 18:30:13.00029447 +0000 UTC m=+882.886267000"
Sep 11 18:30:14.944247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:14.944214    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:30:14.944829 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:14.944807    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:30:16.954053 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:16.954024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:30:18.965773 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:18.965737    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:30:23.072988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.072957    2582 generic.go:356] "Generic (PLEG): container finished" podID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerID="64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc" exitCode=0
Sep 11 18:30:23.073526 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.073037    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerDied","Data":"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"}
Sep 11 18:30:23.073589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.073559    2582 scope.go:117] "RemoveContainer" containerID="64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"
Sep 11 18:30:23.074883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.074849    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:23.074974 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.074920    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:23.074974 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.074957    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:23.990749 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:23.990659    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:24.079025 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.078989    2582 generic.go:356] "Generic (PLEG): container finished" podID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerID="8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4" exitCode=0
Sep 11 18:30:24.079465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.079070    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerDied","Data":"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"}
Sep 11 18:30:24.079534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.079507    2582 scope.go:117] "RemoveContainer" containerID="8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"
Sep 11 18:30:24.080677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.080641    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:24.080761 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.080713    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:24.080761 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.080745    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:24.081135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.081116    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerStarted","Data":"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"}
Sep 11 18:30:24.172328 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.172304    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:30:24.220194 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.219338    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:30:24.220194 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.219477    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.228229 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.223288    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d18cc-openshift-default-dockercfg-rwwx2\""
Sep 11 18:30:24.265954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265791    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.265954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.265954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265850    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.265954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.265954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265909    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.266318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.265986    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.266318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.266076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrbjk\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.266318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.266108    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.266318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.266141    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.366912 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.366865    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrbjk\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.366919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.366947    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.366978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367063    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367477 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367477 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.367577 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367476    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.368038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.368038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.368038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.367982    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.368440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.368358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.370239 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.370210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.370423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.370405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.379137 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.379112    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrbjk\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.380832 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.380815    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token\") pod \"e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") " pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.539479 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.539405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:24.912438 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:24.912385    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46d2127_3530_4de8_9f3e_761834ebde59.slice/crio-91d008ad9dca450ebb7331ffe9c347d402819253596e61b6fff8b56e79ce3d64 WatchSource:0}: Error finding container 91d008ad9dca450ebb7331ffe9c347d402819253596e61b6fff8b56e79ce3d64: Status 404 returned error can't find the container with id 91d008ad9dca450ebb7331ffe9c347d402819253596e61b6fff8b56e79ce3d64
Sep 11 18:30:24.915889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.915783    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:30:24.922044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.920775    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:24.922044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.920864    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:24.922044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:24.920904    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:25.078499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.078463    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:30:25.079967 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.079902    2582 patch_prober.go:28] interesting pod/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Sep 11 18:30:25.080573 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.080538    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Sep 11 18:30:25.091208 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.090471    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerStarted","Data":"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"}
Sep 11 18:30:25.117193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.099411    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerStarted","Data":"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"}
Sep 11 18:30:25.117193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.099499    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerStarted","Data":"91d008ad9dca450ebb7331ffe9c347d402819253596e61b6fff8b56e79ce3d64"}
Sep 11 18:30:25.253582 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.253429    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podStartSLOduration=1.253406825 podStartE2EDuration="1.253406825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:25.252465435 +0000 UTC m=+895.138437925" watchObservedRunningTime="2026-09-11 18:30:25.253406825 +0000 UTC m=+895.139379315"
Sep 11 18:30:25.540144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:25.540026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:26.017665 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.017628    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:30:26.017877 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.017799    2582 patch_prober.go:28] interesting pod/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:30:26.017877 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.017849    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 18:30:26.078813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.078761    2582 patch_prober.go:28] interesting pod/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Sep 11 18:30:26.078980 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.078833    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Sep 11 18:30:26.546589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.546556    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:26.882382 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.882298    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:30:26.882382 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:26.882365    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:30:27.017455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.017408    2582 patch_prober.go:28] interesting pod/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:30:27.017658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.017481    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 18:30:27.079045 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.079003    2582 patch_prober.go:28] interesting pod/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Sep 11 18:30:27.079206 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.079075    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Sep 11 18:30:27.113605 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.113569    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:27.120740 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:27.120708    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:28.016606 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:28.016557    2582 patch_prober.go:28] interesting pod/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:30:28.017159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:28.016631    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 18:30:28.079374 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:28.079334    2582 patch_prober.go:28] interesting pod/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Sep 11 18:30:28.079539 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:28.079447    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Sep 11 18:30:29.017482 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:29.017444    2582 patch_prober.go:28] interesting pod/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:30:29.018074 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:29.018038    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 18:30:29.079186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:29.079145    2582 patch_prober.go:28] interesting pod/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Sep 11 18:30:29.079366 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:29.079297    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Sep 11 18:30:30.017282 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:30.017244    2582 patch_prober.go:28] interesting pod/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused" start-of-body=
Sep 11 18:30:30.017464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:30.017303    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.27:15021/healthz/ready\": dial tcp 10.132.0.27:15021: connect: connection refused"
Sep 11 18:30:30.082934 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:30.082900    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:30:30.083844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:30.083823    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:30:31.020943 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:31.020905    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:30:31.021884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:31.021866    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:30:32.748785 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.748755    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"]
Sep 11 18:30:32.751293 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.751275    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:32.781149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.781121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:30:32.804272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.804240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"]
Sep 11 18:30:32.855933 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.855888    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg\" (UID: \"6c40919f-0e7b-4d88-904d-2706feb5de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:32.957368 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.957334    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg\" (UID: \"6c40919f-0e7b-4d88-904d-2706feb5de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:32.957976 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:32.957955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg\" (UID: \"6c40919f-0e7b-4d88-904d-2706feb5de96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:33.061015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:33.060932    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:33.355328 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:33.355300    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"]
Sep 11 18:30:33.356928 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:33.356902    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c40919f_0e7b_4d88_904d_2706feb5de96.slice/crio-d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef WatchSource:0}: Error finding container d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef: Status 404 returned error can't find the container with id d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef
Sep 11 18:30:34.153695 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:34.153642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg" event={"ID":"6c40919f-0e7b-4d88-904d-2706feb5de96","Type":"ContainerStarted","Data":"d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef"}
Sep 11 18:30:36.168793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:36.168700    2582 generic.go:356] "Generic (PLEG): container finished" podID="6c40919f-0e7b-4d88-904d-2706feb5de96" containerID="d634e22c39847c930b20662aa46f9ebdef3a7e49d88ede58331804986d60ca83" exitCode=0
Sep 11 18:30:36.168793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:36.168769    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg" event={"ID":"6c40919f-0e7b-4d88-904d-2706feb5de96","Type":"ContainerDied","Data":"d634e22c39847c930b20662aa46f9ebdef3a7e49d88ede58331804986d60ca83"}
Sep 11 18:30:37.315983 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:37.315958    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:37.398532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:37.398491    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca\") pod \"6c40919f-0e7b-4d88-904d-2706feb5de96\" (UID: \"6c40919f-0e7b-4d88-904d-2706feb5de96\") "
Sep 11 18:30:37.398861 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:37.398837    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c40919f-0e7b-4d88-904d-2706feb5de96" (UID: "6c40919f-0e7b-4d88-904d-2706feb5de96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:37.500225 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:37.500130    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c40919f-0e7b-4d88-904d-2706feb5de96-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:30:38.179739 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:38.179710    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"
Sep 11 18:30:38.179739 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:38.179736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg" event={"ID":"6c40919f-0e7b-4d88-904d-2706feb5de96","Type":"ContainerDied","Data":"d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef"}
Sep 11 18:30:38.179992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:38.179776    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d51efa925808a9465b8fa66f08762c60eb3ccbef01ffc9c7d07aa7ecbc8c2cef"
Sep 11 18:30:42.236724 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.236688    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:42.237126 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.236917    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" podUID="dbdc3480-57c9-4419-b185-51546d065e02" containerName="payload-pre-processing" containerID="cri-o://bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f" gracePeriod=30
Sep 11 18:30:42.454090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.454056    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"]
Sep 11 18:30:42.457375 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.457345    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aa3ba758-t99wg"]
Sep 11 18:30:42.500636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.500569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:42.546310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.546274    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jwt2\" (UniqueName: \"kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2\") pod \"dbdc3480-57c9-4419-b185-51546d065e02\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") "
Sep 11 18:30:42.546478 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.546439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume\") pod \"dbdc3480-57c9-4419-b185-51546d065e02\" (UID: \"dbdc3480-57c9-4419-b185-51546d065e02\") "
Sep 11 18:30:42.546777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.546754    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbdc3480-57c9-4419-b185-51546d065e02" (UID: "dbdc3480-57c9-4419-b185-51546d065e02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:30:42.548707 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.548674    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2" (OuterVolumeSpecName: "kube-api-access-4jwt2") pod "dbdc3480-57c9-4419-b185-51546d065e02" (UID: "dbdc3480-57c9-4419-b185-51546d065e02"). InnerVolumeSpecName "kube-api-access-4jwt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:30:42.555405 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.555383    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c40919f-0e7b-4d88-904d-2706feb5de96" path="/var/lib/kubelet/pods/6c40919f-0e7b-4d88-904d-2706feb5de96/volumes"
Sep 11 18:30:42.647569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.647534    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwt2\" (UniqueName: \"kubernetes.io/projected/dbdc3480-57c9-4419-b185-51546d065e02-kube-api-access-4jwt2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:30:42.647569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:42.647562    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdc3480-57c9-4419-b185-51546d065e02-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:30:43.207237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.207201    2582 generic.go:356] "Generic (PLEG): container finished" podID="dbdc3480-57c9-4419-b185-51546d065e02" containerID="bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f" exitCode=0
Sep 11 18:30:43.207237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.207238    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"
Sep 11 18:30:43.207504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.207276    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" event={"ID":"dbdc3480-57c9-4419-b185-51546d065e02","Type":"ContainerDied","Data":"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"}
Sep 11 18:30:43.207504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.207339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w" event={"ID":"dbdc3480-57c9-4419-b185-51546d065e02","Type":"ContainerDied","Data":"61f0e45229f3e3a69b293bc7bf5438b1b550c099f2cdcf73eb5966a5d2119775"}
Sep 11 18:30:43.207504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.207365    2582 scope.go:117] "RemoveContainer" containerID="bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"
Sep 11 18:30:43.216839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.216775    2582 scope.go:117] "RemoveContainer" containerID="bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"
Sep 11 18:30:43.217102 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:30:43.217084    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f\": container with ID starting with bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f not found: ID does not exist" containerID="bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"
Sep 11 18:30:43.217203 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.217110    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f"} err="failed to get container status \"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f\": rpc error: code = NotFound desc = could not find container \"bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f\": container with ID starting with bd5db39f8a99670bb07679b4e810c05fb131b1d1cae2e834aa65954c9e2a9e2f not found: ID does not exist"
Sep 11 18:30:43.238286 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.238226    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:43.240411 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:43.240388    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aa3ba758-584d877898-n4t8w"]
Sep 11 18:30:44.553711 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:44.553676    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdc3480-57c9-4419-b185-51546d065e02" path="/var/lib/kubelet/pods/dbdc3480-57c9-4419-b185-51546d065e02/volumes"
Sep 11 18:30:49.109942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.109898    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:30:49.111015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.110988    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdc3480-57c9-4419-b185-51546d065e02" containerName="payload-pre-processing"
Sep 11 18:30:49.111183 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.111151    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdc3480-57c9-4419-b185-51546d065e02" containerName="payload-pre-processing"
Sep 11 18:30:49.111315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.111301    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c40919f-0e7b-4d88-904d-2706feb5de96" containerName="revoke-keys"
Sep 11 18:30:49.111397 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.111388    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c40919f-0e7b-4d88-904d-2706feb5de96" containerName="revoke-keys"
Sep 11 18:30:49.111597 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.111585    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdc3480-57c9-4419-b185-51546d065e02" containerName="payload-pre-processing"
Sep 11 18:30:49.111685 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.111675    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c40919f-0e7b-4d88-904d-2706feb5de96" containerName="revoke-keys"
Sep 11 18:30:49.116346 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.116323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.119560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.119528    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-0aacf4f7-serving-cert\""
Sep 11 18:30:49.125561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.125408    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:30:49.221376 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.221333    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.221534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.221395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.221534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.221453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.221534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.221499    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf92r\" (UniqueName: \"kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.322315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.322272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.322486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.322328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.322486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.322358    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf92r\" (UniqueName: \"kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.322486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.322422    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.322973 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:30:49.322949    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-serving-cert: secret "maas-api-e2e-migrate-0aacf4f7-serving-cert" not found
Sep 11 18:30:49.323067 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:30:49.323055    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls podName:c5e243bc-848a-4d27-8c53-96763dc95834 nodeName:}" failed. No retries permitted until 2026-09-11 18:30:49.823032168 +0000 UTC m=+919.709004643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls") pod "maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" (UID: "c5e243bc-848a-4d27-8c53-96763dc95834") : secret "maas-api-e2e-migrate-0aacf4f7-serving-cert" not found
Sep 11 18:30:49.323492 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.323474    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.324811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.324795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.331895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.331866    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf92r\" (UniqueName: \"kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.813467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.811626    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:30:49.814682 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.814658    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:49.818247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.818217    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-0aacf4f7-dockercfg-rkmnv\""
Sep 11 18:30:49.818364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.818289    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-0aacf4f7\""
Sep 11 18:30:49.827580 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.827563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.830237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.830215    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") pod \"maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:49.831364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.831342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:30:49.868406 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.868378    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:30:49.870756 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.870737    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:49.882520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.881837    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:30:49.930725 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.930688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:49.931068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:49.931039    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7cf2\" (UniqueName: \"kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.032763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.032728    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.032937 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.032791    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7cf2\" (UniqueName: \"kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.032937 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.032820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.032937 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.032840    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsd59\" (UniqueName: \"kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.032937 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.032934    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:50.033853 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.033827    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.041579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.041557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7cf2\" (UniqueName: \"kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2\") pod \"payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") " pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.128493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.128399    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:50.139532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.135024    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.139532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.135076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsd59\" (UniqueName: \"kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.139532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.135959    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.144700 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.144500    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsd59\" (UniqueName: \"kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59\") pod \"payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.184513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.184476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:50.342569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.342531    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:30:50.345383 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:50.345348    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e243bc_848a_4d27_8c53_96763dc95834.slice/crio-8b61056e8f52eaef3e9e5aa0271ac5b59652734959f6fbfc04d1588a05c07f69 WatchSource:0}: Error finding container 8b61056e8f52eaef3e9e5aa0271ac5b59652734959f6fbfc04d1588a05c07f69: Status 404 returned error can't find the container with id 8b61056e8f52eaef3e9e5aa0271ac5b59652734959f6fbfc04d1588a05c07f69
Sep 11 18:30:50.424061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.424023    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:30:50.426513 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:50.426485    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8151cbe6_a4a2_459d_b740_10f905da7645.slice/crio-be834a8f6a84a66fcaf14a70135bf5c78546a9e552e8d0ffad5c5b0a29203967 WatchSource:0}: Error finding container be834a8f6a84a66fcaf14a70135bf5c78546a9e552e8d0ffad5c5b0a29203967: Status 404 returned error can't find the container with id be834a8f6a84a66fcaf14a70135bf5c78546a9e552e8d0ffad5c5b0a29203967
Sep 11 18:30:50.501190 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:30:50.501135    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf128993d_1ad6_468c_98a6_a119dc18b158.slice/crio-ce3c9406e88c185928a6126cbfa4fe4619ed255c1c184dfb7edc88651c1aad00 WatchSource:0}: Error finding container ce3c9406e88c185928a6126cbfa4fe4619ed255c1c184dfb7edc88651c1aad00: Status 404 returned error can't find the container with id ce3c9406e88c185928a6126cbfa4fe4619ed255c1c184dfb7edc88651c1aad00
Sep 11 18:30:50.502355 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:50.502327    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:30:51.252521 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.252462    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" event={"ID":"8151cbe6-a4a2-459d-b740-10f905da7645","Type":"ContainerStarted","Data":"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"}
Sep 11 18:30:51.253024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.252536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:30:51.253024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.252558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" event={"ID":"8151cbe6-a4a2-459d-b740-10f905da7645","Type":"ContainerStarted","Data":"be834a8f6a84a66fcaf14a70135bf5c78546a9e552e8d0ffad5c5b0a29203967"}
Sep 11 18:30:51.254335 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.254302    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" event={"ID":"c5e243bc-848a-4d27-8c53-96763dc95834","Type":"ContainerStarted","Data":"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"}
Sep 11 18:30:51.254505 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.254362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:51.254505 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.254384    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" event={"ID":"c5e243bc-848a-4d27-8c53-96763dc95834","Type":"ContainerStarted","Data":"8b61056e8f52eaef3e9e5aa0271ac5b59652734959f6fbfc04d1588a05c07f69"}
Sep 11 18:30:51.256071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.256035    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" event={"ID":"f128993d-1ad6-468c-98a6-a119dc18b158","Type":"ContainerStarted","Data":"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"}
Sep 11 18:30:51.256071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.256073    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" event={"ID":"f128993d-1ad6-468c-98a6-a119dc18b158","Type":"ContainerStarted","Data":"ce3c9406e88c185928a6126cbfa4fe4619ed255c1c184dfb7edc88651c1aad00"}
Sep 11 18:30:51.256274 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.256131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:30:51.276793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.276743    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" podStartSLOduration=2.276729276 podStartE2EDuration="2.276729276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:51.274820069 +0000 UTC m=+921.160792559" watchObservedRunningTime="2026-09-11 18:30:51.276729276 +0000 UTC m=+921.162701765"
Sep 11 18:30:51.310728 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:51.310678    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" podStartSLOduration=2.310658408 podStartE2EDuration="2.310658408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:51.308022534 +0000 UTC m=+921.193995031" watchObservedRunningTime="2026-09-11 18:30:51.310658408 +0000 UTC m=+921.196630898"
Sep 11 18:30:54.273621 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.273587    2582 generic.go:356] "Generic (PLEG): container finished" podID="b46d2127-3530-4de8-9f3e-761834ebde59" containerID="2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19" exitCode=0
Sep 11 18:30:54.274105 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.273670    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerDied","Data":"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"}
Sep 11 18:30:54.274105 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.274061    2582 scope.go:117] "RemoveContainer" containerID="2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"
Sep 11 18:30:54.275264 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.275226    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:54.275372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.275301    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:54.275372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.275333    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:30:54.330600 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:54.330553    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" podStartSLOduration=4.995557729 podStartE2EDuration="5.330535661s" totalImagesPullingTime="334.977932ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:30:51.332955371 +0000 UTC m=+921.218927862" watchObservedRunningTime="2026-09-11 18:30:54.330535661 +0000 UTC m=+924.216508149"
Sep 11 18:30:55.280742 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:55.280705    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerStarted","Data":"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"}
Sep 11 18:30:55.540631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:55.540553    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:55.541475 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:55.541451    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:30:55.541531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:55.541503    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:30:56.540605 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:56.540568    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:30:56.540982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:56.540618    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:30:57.113645 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:57.113614    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:30:57.267440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:57.267411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:30:57.540420 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:57.540385    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:30:57.540655 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:57.540629    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:30:58.540571 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:58.540529    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:30:58.540738 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:58.540588    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:30:59.540622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:59.540566    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:30:59.540796 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:30:59.540631    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:31:00.540630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:00.540595    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body=
Sep 11 18:31:00.540833 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:00.540651    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused"
Sep 11 18:31:01.544164 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:01.544134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:31:01.545036 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:01.545017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:31:02.263655 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:02.263624    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:31:02.263848 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:02.263785    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:31:11.815384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.815345    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:31:11.815875 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.815579    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" podUID="c5e243bc-848a-4d27-8c53-96763dc95834" containerName="maas-api" containerID="cri-o://380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57" gracePeriod=30
Sep 11 18:31:11.906012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.905963    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:31:11.906369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.906252    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" podUID="8151cbe6-a4a2-459d-b740-10f905da7645" containerName="payload-processing" containerID="cri-o://b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5" gracePeriod=30
Sep 11 18:31:11.910090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.909639    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:31:11.910090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:11.909912    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" podUID="f128993d-1ad6-468c-98a6-a119dc18b158" containerName="payload-pre-processing" containerID="cri-o://bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a" gracePeriod=30
Sep 11 18:31:12.162942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.162846    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:31:12.215096 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.214595    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:31:12.220118 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.220091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs\") pod \"c5e243bc-848a-4d27-8c53-96763dc95834\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238195    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf92r\" (UniqueName: \"kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r\") pod \"c5e243bc-848a-4d27-8c53-96763dc95834\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") pod \"c5e243bc-848a-4d27-8c53-96763dc95834\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238338    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume\") pod \"8151cbe6-a4a2-459d-b740-10f905da7645\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238398    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca\") pod \"c5e243bc-848a-4d27-8c53-96763dc95834\" (UID: \"c5e243bc-848a-4d27-8c53-96763dc95834\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238427    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7cf2\" (UniqueName: \"kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2\") pod \"8151cbe6-a4a2-459d-b740-10f905da7645\" (UID: \"8151cbe6-a4a2-459d-b740-10f905da7645\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238459    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume\") pod \"f128993d-1ad6-468c-98a6-a119dc18b158\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238500    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsd59\" (UniqueName: \"kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59\") pod \"f128993d-1ad6-468c-98a6-a119dc18b158\" (UID: \"f128993d-1ad6-468c-98a6-a119dc18b158\") "
Sep 11 18:31:12.238945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.238878    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8151cbe6-a4a2-459d-b740-10f905da7645" (UID: "8151cbe6-a4a2-459d-b740-10f905da7645"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:12.240643 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.240617    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f128993d-1ad6-468c-98a6-a119dc18b158" (UID: "f128993d-1ad6-468c-98a6-a119dc18b158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:12.242453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.241049    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5e243bc-848a-4d27-8c53-96763dc95834" (UID: "c5e243bc-848a-4d27-8c53-96763dc95834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:12.245145 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.245093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5e243bc-848a-4d27-8c53-96763dc95834" (UID: "c5e243bc-848a-4d27-8c53-96763dc95834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:31:12.245488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.245374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2" (OuterVolumeSpecName: "kube-api-access-k7cf2") pod "8151cbe6-a4a2-459d-b740-10f905da7645" (UID: "8151cbe6-a4a2-459d-b740-10f905da7645"). InnerVolumeSpecName "kube-api-access-k7cf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:12.245488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.245389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r" (OuterVolumeSpecName: "kube-api-access-wf92r") pod "c5e243bc-848a-4d27-8c53-96763dc95834" (UID: "c5e243bc-848a-4d27-8c53-96763dc95834"). InnerVolumeSpecName "kube-api-access-wf92r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:12.246135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.246099    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c5e243bc-848a-4d27-8c53-96763dc95834" (UID: "c5e243bc-848a-4d27-8c53-96763dc95834"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:31:12.247522 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.247440    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59" (OuterVolumeSpecName: "kube-api-access-xsd59") pod "f128993d-1ad6-468c-98a6-a119dc18b158" (UID: "f128993d-1ad6-468c-98a6-a119dc18b158"). InnerVolumeSpecName "kube-api-access-xsd59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340271    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8151cbe6-a4a2-459d-b740-10f905da7645-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340305    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e243bc-848a-4d27-8c53-96763dc95834-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340315    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7cf2\" (UniqueName: \"kubernetes.io/projected/8151cbe6-a4a2-459d-b740-10f905da7645-kube-api-access-k7cf2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340329    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f128993d-1ad6-468c-98a6-a119dc18b158-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340343    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsd59\" (UniqueName: \"kubernetes.io/projected/f128993d-1ad6-468c-98a6-a119dc18b158-kube-api-access-xsd59\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340356    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340365    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf92r\" (UniqueName: \"kubernetes.io/projected/c5e243bc-848a-4d27-8c53-96763dc95834-kube-api-access-wf92r\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.340367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.340373    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5e243bc-848a-4d27-8c53-96763dc95834-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:12.383080 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.383046    2582 generic.go:356] "Generic (PLEG): container finished" podID="8151cbe6-a4a2-459d-b740-10f905da7645" containerID="b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5" exitCode=0
Sep 11 18:31:12.383253 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.383101    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"
Sep 11 18:31:12.383253 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.383099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" event={"ID":"8151cbe6-a4a2-459d-b740-10f905da7645","Type":"ContainerDied","Data":"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"}
Sep 11 18:31:12.383253 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.383222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d" event={"ID":"8151cbe6-a4a2-459d-b740-10f905da7645","Type":"ContainerDied","Data":"be834a8f6a84a66fcaf14a70135bf5c78546a9e552e8d0ffad5c5b0a29203967"}
Sep 11 18:31:12.383438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.383256    2582 scope.go:117] "RemoveContainer" containerID="b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"
Sep 11 18:31:12.384941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.384919    2582 generic.go:356] "Generic (PLEG): container finished" podID="c5e243bc-848a-4d27-8c53-96763dc95834" containerID="380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57" exitCode=0
Sep 11 18:31:12.385042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.384979    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"
Sep 11 18:31:12.385042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.385004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" event={"ID":"c5e243bc-848a-4d27-8c53-96763dc95834","Type":"ContainerDied","Data":"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"}
Sep 11 18:31:12.385142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.385052    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc" event={"ID":"c5e243bc-848a-4d27-8c53-96763dc95834","Type":"ContainerDied","Data":"8b61056e8f52eaef3e9e5aa0271ac5b59652734959f6fbfc04d1588a05c07f69"}
Sep 11 18:31:12.388430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.386550    2582 generic.go:356] "Generic (PLEG): container finished" podID="f128993d-1ad6-468c-98a6-a119dc18b158" containerID="bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a" exitCode=0
Sep 11 18:31:12.388430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.386627    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" event={"ID":"f128993d-1ad6-468c-98a6-a119dc18b158","Type":"ContainerDied","Data":"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"}
Sep 11 18:31:12.388430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.386699    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"
Sep 11 18:31:12.388810 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.388788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv" event={"ID":"f128993d-1ad6-468c-98a6-a119dc18b158","Type":"ContainerDied","Data":"ce3c9406e88c185928a6126cbfa4fe4619ed255c1c184dfb7edc88651c1aad00"}
Sep 11 18:31:12.395084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.395064    2582 scope.go:117] "RemoveContainer" containerID="b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"
Sep 11 18:31:12.395445 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:12.395419    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5\": container with ID starting with b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5 not found: ID does not exist" containerID="b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"
Sep 11 18:31:12.395520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.395457    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5"} err="failed to get container status \"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5\": rpc error: code = NotFound desc = could not find container \"b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5\": container with ID starting with b5e3c6cbb54f367b53706dfb02cf4b1e5c31845577c88c0360c2d737f65c1ff5 not found: ID does not exist"
Sep 11 18:31:12.395520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.395480    2582 scope.go:117] "RemoveContainer" containerID="380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"
Sep 11 18:31:12.404984 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.404969    2582 scope.go:117] "RemoveContainer" containerID="380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"
Sep 11 18:31:12.405247 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:12.405228    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57\": container with ID starting with 380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57 not found: ID does not exist" containerID="380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"
Sep 11 18:31:12.405316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.405253    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57"} err="failed to get container status \"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57\": rpc error: code = NotFound desc = could not find container \"380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57\": container with ID starting with 380ddc146bec1bf01d5c7a759635046656125c112dc2edbf2877254bd849bd57 not found: ID does not exist"
Sep 11 18:31:12.405316 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.405268    2582 scope.go:117] "RemoveContainer" containerID="bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"
Sep 11 18:31:12.409921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.409904    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:31:12.413590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.413574    2582 scope.go:117] "RemoveContainer" containerID="bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"
Sep 11 18:31:12.413858 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:12.413840    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a\": container with ID starting with bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a not found: ID does not exist" containerID="bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"
Sep 11 18:31:12.413928 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.413865    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a"} err="failed to get container status \"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a\": rpc error: code = NotFound desc = could not find container \"bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a\": container with ID starting with bef6eee95b6ecb280daea2ec7caf1e4d0a364dd5ebcb754cfcef28686ef66e5a not found: ID does not exist"
Sep 11 18:31:12.414119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.414101    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-0aacf4f7-5c6b548846-75v6d"]
Sep 11 18:31:12.425780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.425758    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:31:12.428880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.428860    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-0aacf4f7-7b557fdd65-mfjhc"]
Sep 11 18:31:12.439074 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.439055    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:31:12.445782 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.445763    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-0aacf4f7-9c594c8fd-gd4rv"]
Sep 11 18:31:12.554421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.554391    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8151cbe6-a4a2-459d-b740-10f905da7645" path="/var/lib/kubelet/pods/8151cbe6-a4a2-459d-b740-10f905da7645/volumes"
Sep 11 18:31:12.554808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.554794    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e243bc-848a-4d27-8c53-96763dc95834" path="/var/lib/kubelet/pods/c5e243bc-848a-4d27-8c53-96763dc95834/volumes"
Sep 11 18:31:12.555113 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:12.555102    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f128993d-1ad6-468c-98a6-a119dc18b158" path="/var/lib/kubelet/pods/f128993d-1ad6-468c-98a6-a119dc18b158/volumes"
Sep 11 18:31:18.513376 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513339    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513724    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f128993d-1ad6-468c-98a6-a119dc18b158" containerName="payload-pre-processing"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513736    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f128993d-1ad6-468c-98a6-a119dc18b158" containerName="payload-pre-processing"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513746    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8151cbe6-a4a2-459d-b740-10f905da7645" containerName="payload-processing"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513752    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8151cbe6-a4a2-459d-b740-10f905da7645" containerName="payload-processing"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513774    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e243bc-848a-4d27-8c53-96763dc95834" containerName="maas-api"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513780    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e243bc-848a-4d27-8c53-96763dc95834" containerName="maas-api"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513833    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f128993d-1ad6-468c-98a6-a119dc18b158" containerName="payload-pre-processing"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513841    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e243bc-848a-4d27-8c53-96763dc95834" containerName="maas-api"
Sep 11 18:31:18.513839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.513848    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8151cbe6-a4a2-459d-b740-10f905da7645" containerName="payload-processing"
Sep 11 18:31:18.518542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.518520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.521422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.521400    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1fc84f74-serving-cert\""
Sep 11 18:31:18.531659 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.531635    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:18.597596 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.597564    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5qpg\" (UniqueName: \"kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.597815 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.597783    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.597815 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.597834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.598055 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.597889    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.699123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.699084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5qpg\" (UniqueName: \"kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.699341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.699151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.699341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.699199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.699341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.699239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.699941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.699916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.701641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.701622    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.701879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.701857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.707944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.707916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5qpg\" (UniqueName: \"kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg\") pod \"maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:18.829149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:18.829072    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:19.109364 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.109340    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:19.111219 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:19.111187    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883955eb_dc65_4ad0_8a98_5fae25959e13.slice/crio-852aef26edb15fb877b93b9893be68d10c700fade1ac78e5fcf0029a0d9f44af WatchSource:0}: Error finding container 852aef26edb15fb877b93b9893be68d10c700fade1ac78e5fcf0029a0d9f44af: Status 404 returned error can't find the container with id 852aef26edb15fb877b93b9893be68d10c700fade1ac78e5fcf0029a0d9f44af
Sep 11 18:31:19.214915 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.214881    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:19.219293 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.219272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.222012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.221991    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1fc84f74-dockercfg-hf954\""
Sep 11 18:31:19.222012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.221993    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1fc84f74\""
Sep 11 18:31:19.228151 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.228129    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:19.254490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.254452    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:19.257630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.257610    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.272596 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.272570    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:19.304226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.304191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nlmd\" (UniqueName: \"kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.304407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.304244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2vtd\" (UniqueName: \"kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.304407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.304320    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.304407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.304360    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.405439 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.405409    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nlmd\" (UniqueName: \"kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.405613 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.405449    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2vtd\" (UniqueName: \"kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.405613 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.405516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.405613 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.405551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.406229 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.406209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.406297 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.406233    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.415570 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.415550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2vtd\" (UniqueName: \"kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd\") pod \"payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.415662 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.415606    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nlmd\" (UniqueName: \"kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd\") pod \"payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") " pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.428736 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.428711    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" event={"ID":"883955eb-dc65-4ad0-8a98-5fae25959e13","Type":"ContainerStarted","Data":"852aef26edb15fb877b93b9893be68d10c700fade1ac78e5fcf0029a0d9f44af"}
Sep 11 18:31:19.531536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.531516    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:19.568653 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.568581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:19.804351 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.804326    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:19.806239 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:19.806208    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0aa6d43_c579_435e_90b2_76e68d1f721c.slice/crio-73457ccd29e0fe3c60de73b38c05936ad979ec90d3d2d943cc5e7b9fb9b0e1f6 WatchSource:0}: Error finding container 73457ccd29e0fe3c60de73b38c05936ad979ec90d3d2d943cc5e7b9fb9b0e1f6: Status 404 returned error can't find the container with id 73457ccd29e0fe3c60de73b38c05936ad979ec90d3d2d943cc5e7b9fb9b0e1f6
Sep 11 18:31:19.858453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:19.858426    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:19.861630 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:19.861604    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f099d93_d444_469c_a51f_dda2aea71ce2.slice/crio-4f892f1c4b58c61d8b992c3163d98f640d85742e7684bbece6e5a6ca2f5019ba WatchSource:0}: Error finding container 4f892f1c4b58c61d8b992c3163d98f640d85742e7684bbece6e5a6ca2f5019ba: Status 404 returned error can't find the container with id 4f892f1c4b58c61d8b992c3163d98f640d85742e7684bbece6e5a6ca2f5019ba
Sep 11 18:31:20.435611 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.435571    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" event={"ID":"0f099d93-d444-469c-a51f-dda2aea71ce2","Type":"ContainerStarted","Data":"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"}
Sep 11 18:31:20.435813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.435669    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" event={"ID":"0f099d93-d444-469c-a51f-dda2aea71ce2","Type":"ContainerStarted","Data":"4f892f1c4b58c61d8b992c3163d98f640d85742e7684bbece6e5a6ca2f5019ba"}
Sep 11 18:31:20.435813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.435701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:20.437356 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.437330    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" event={"ID":"e0aa6d43-c579-435e-90b2-76e68d1f721c","Type":"ContainerStarted","Data":"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"}
Sep 11 18:31:20.437460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.437374    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" event={"ID":"e0aa6d43-c579-435e-90b2-76e68d1f721c","Type":"ContainerStarted","Data":"73457ccd29e0fe3c60de73b38c05936ad979ec90d3d2d943cc5e7b9fb9b0e1f6"}
Sep 11 18:31:20.437460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.437434    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:20.438803 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.438786    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" event={"ID":"883955eb-dc65-4ad0-8a98-5fae25959e13","Type":"ContainerStarted","Data":"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"}
Sep 11 18:31:20.438931 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.438915    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:20.452989 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.452950    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" podStartSLOduration=1.45293632 podStartE2EDuration="1.45293632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:20.451970734 +0000 UTC m=+950.337943238" watchObservedRunningTime="2026-09-11 18:31:20.45293632 +0000 UTC m=+950.338908809"
Sep 11 18:31:20.469495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.469450    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" podStartSLOduration=1.469436713 podStartE2EDuration="1.469436713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:20.468304216 +0000 UTC m=+950.354276710" watchObservedRunningTime="2026-09-11 18:31:20.469436713 +0000 UTC m=+950.355409201"
Sep 11 18:31:20.487404 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:20.487344    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" podStartSLOduration=2.117337876 podStartE2EDuration="2.487324839s" totalImagesPullingTime="369.986963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:20.486529252 +0000 UTC m=+950.372501742" watchObservedRunningTime="2026-09-11 18:31:20.487324839 +0000 UTC m=+950.373297330"
Sep 11 18:31:26.449282 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:26.449253    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:31.446005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:31.445976    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:31.446406 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:31.446045    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:47.750550 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.750512    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:47.750977 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.750774    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" podUID="883955eb-dc65-4ad0-8a98-5fae25959e13" containerName="maas-api" containerID="cri-o://1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652" gracePeriod=30
Sep 11 18:31:47.810339 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.810309    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:47.810560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.810537    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" podUID="e0aa6d43-c579-435e-90b2-76e68d1f721c" containerName="payload-processing" containerID="cri-o://c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4" gracePeriod=30
Sep 11 18:31:47.816055 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.816026    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:47.816424 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:47.816383    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" podUID="0f099d93-d444-469c-a51f-dda2aea71ce2" containerName="payload-pre-processing" containerID="cri-o://886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf" gracePeriod=30
Sep 11 18:31:48.118963 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.118937    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:48.121926 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.121903    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:48.131464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.131447    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:48.277236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca\") pod \"883955eb-dc65-4ad0-8a98-5fae25959e13\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") "
Sep 11 18:31:48.277377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277255    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs\") pod \"883955eb-dc65-4ad0-8a98-5fae25959e13\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") "
Sep 11 18:31:48.277377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277296    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume\") pod \"0f099d93-d444-469c-a51f-dda2aea71ce2\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") "
Sep 11 18:31:48.277377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277337    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nlmd\" (UniqueName: \"kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd\") pod \"e0aa6d43-c579-435e-90b2-76e68d1f721c\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") "
Sep 11 18:31:48.277377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277376    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls\") pod \"883955eb-dc65-4ad0-8a98-5fae25959e13\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") "
Sep 11 18:31:48.277554 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277394    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2vtd\" (UniqueName: \"kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd\") pod \"0f099d93-d444-469c-a51f-dda2aea71ce2\" (UID: \"0f099d93-d444-469c-a51f-dda2aea71ce2\") "
Sep 11 18:31:48.277554 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277412    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume\") pod \"e0aa6d43-c579-435e-90b2-76e68d1f721c\" (UID: \"e0aa6d43-c579-435e-90b2-76e68d1f721c\") "
Sep 11 18:31:48.277554 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5qpg\" (UniqueName: \"kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg\") pod \"883955eb-dc65-4ad0-8a98-5fae25959e13\" (UID: \"883955eb-dc65-4ad0-8a98-5fae25959e13\") "
Sep 11 18:31:48.277554 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277515    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "883955eb-dc65-4ad0-8a98-5fae25959e13" (UID: "883955eb-dc65-4ad0-8a98-5fae25959e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:48.277753 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277692    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883955eb-dc65-4ad0-8a98-5fae25959e13-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.277753 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.277726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0f099d93-d444-469c-a51f-dda2aea71ce2" (UID: "0f099d93-d444-469c-a51f-dda2aea71ce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:48.278080 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.278052    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0aa6d43-c579-435e-90b2-76e68d1f721c" (UID: "e0aa6d43-c579-435e-90b2-76e68d1f721c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:31:48.279599 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.279570    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd" (OuterVolumeSpecName: "kube-api-access-8nlmd") pod "e0aa6d43-c579-435e-90b2-76e68d1f721c" (UID: "e0aa6d43-c579-435e-90b2-76e68d1f721c"). InnerVolumeSpecName "kube-api-access-8nlmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:48.279686 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.279627    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg" (OuterVolumeSpecName: "kube-api-access-s5qpg") pod "883955eb-dc65-4ad0-8a98-5fae25959e13" (UID: "883955eb-dc65-4ad0-8a98-5fae25959e13"). InnerVolumeSpecName "kube-api-access-s5qpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:48.279836 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.279813    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd" (OuterVolumeSpecName: "kube-api-access-k2vtd") pod "0f099d93-d444-469c-a51f-dda2aea71ce2" (UID: "0f099d93-d444-469c-a51f-dda2aea71ce2"). InnerVolumeSpecName "kube-api-access-k2vtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:31:48.279929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.279910    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "883955eb-dc65-4ad0-8a98-5fae25959e13" (UID: "883955eb-dc65-4ad0-8a98-5fae25959e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:31:48.279968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.279921    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "883955eb-dc65-4ad0-8a98-5fae25959e13" (UID: "883955eb-dc65-4ad0-8a98-5fae25959e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:31:48.378616 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378585    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378616 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378609    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f099d93-d444-469c-a51f-dda2aea71ce2-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378616 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378621    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nlmd\" (UniqueName: \"kubernetes.io/projected/e0aa6d43-c579-435e-90b2-76e68d1f721c-kube-api-access-8nlmd\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378630    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883955eb-dc65-4ad0-8a98-5fae25959e13-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378639    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2vtd\" (UniqueName: \"kubernetes.io/projected/0f099d93-d444-469c-a51f-dda2aea71ce2-kube-api-access-k2vtd\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378646    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0aa6d43-c579-435e-90b2-76e68d1f721c-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.378805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.378655    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5qpg\" (UniqueName: \"kubernetes.io/projected/883955eb-dc65-4ad0-8a98-5fae25959e13-kube-api-access-s5qpg\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:31:48.582780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.582747    2582 generic.go:356] "Generic (PLEG): container finished" podID="0f099d93-d444-469c-a51f-dda2aea71ce2" containerID="886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf" exitCode=0
Sep 11 18:31:48.582887 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.582783    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"
Sep 11 18:31:48.582887 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.582823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" event={"ID":"0f099d93-d444-469c-a51f-dda2aea71ce2","Type":"ContainerDied","Data":"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"}
Sep 11 18:31:48.582887 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.582873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv" event={"ID":"0f099d93-d444-469c-a51f-dda2aea71ce2","Type":"ContainerDied","Data":"4f892f1c4b58c61d8b992c3163d98f640d85742e7684bbece6e5a6ca2f5019ba"}
Sep 11 18:31:48.583054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.582901    2582 scope.go:117] "RemoveContainer" containerID="886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"
Sep 11 18:31:48.584324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.584297    2582 generic.go:356] "Generic (PLEG): container finished" podID="e0aa6d43-c579-435e-90b2-76e68d1f721c" containerID="c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4" exitCode=0
Sep 11 18:31:48.584434 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.584330    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"
Sep 11 18:31:48.584434 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.584379    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" event={"ID":"e0aa6d43-c579-435e-90b2-76e68d1f721c","Type":"ContainerDied","Data":"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"}
Sep 11 18:31:48.584551 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.584444    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld" event={"ID":"e0aa6d43-c579-435e-90b2-76e68d1f721c","Type":"ContainerDied","Data":"73457ccd29e0fe3c60de73b38c05936ad979ec90d3d2d943cc5e7b9fb9b0e1f6"}
Sep 11 18:31:48.586314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.586201    2582 generic.go:356] "Generic (PLEG): container finished" podID="883955eb-dc65-4ad0-8a98-5fae25959e13" containerID="1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652" exitCode=0
Sep 11 18:31:48.586314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.586239    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" event={"ID":"883955eb-dc65-4ad0-8a98-5fae25959e13","Type":"ContainerDied","Data":"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"}
Sep 11 18:31:48.586314 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.586252    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"
Sep 11 18:31:48.586549 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.586280    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8" event={"ID":"883955eb-dc65-4ad0-8a98-5fae25959e13","Type":"ContainerDied","Data":"852aef26edb15fb877b93b9893be68d10c700fade1ac78e5fcf0029a0d9f44af"}
Sep 11 18:31:48.596344 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.596320    2582 scope.go:117] "RemoveContainer" containerID="886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"
Sep 11 18:31:48.596664 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:48.596643    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf\": container with ID starting with 886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf not found: ID does not exist" containerID="886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"
Sep 11 18:31:48.596742 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.596675    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf"} err="failed to get container status \"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf\": rpc error: code = NotFound desc = could not find container \"886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf\": container with ID starting with 886a5bcf15ae1c55ffacd147225cc8fda363f7b1ed03de8a1560937d819fafdf not found: ID does not exist"
Sep 11 18:31:48.596742 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.596696    2582 scope.go:117] "RemoveContainer" containerID="c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"
Sep 11 18:31:48.604458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.604438    2582 scope.go:117] "RemoveContainer" containerID="c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"
Sep 11 18:31:48.604736 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:48.604720    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4\": container with ID starting with c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4 not found: ID does not exist" containerID="c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"
Sep 11 18:31:48.604792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.604742    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4"} err="failed to get container status \"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4\": rpc error: code = NotFound desc = could not find container \"c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4\": container with ID starting with c38ab856fffe2445f397e73ddc7a899fd717f636bc3289fcc2ecefa3a63559a4 not found: ID does not exist"
Sep 11 18:31:48.604792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.604756    2582 scope.go:117] "RemoveContainer" containerID="1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"
Sep 11 18:31:48.606009 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.605975    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:48.607760 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.607742    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1fc84f74-78c9c94d58-ls4sv"]
Sep 11 18:31:48.612704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.612689    2582 scope.go:117] "RemoveContainer" containerID="1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"
Sep 11 18:31:48.612952 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:31:48.612933    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652\": container with ID starting with 1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652 not found: ID does not exist" containerID="1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"
Sep 11 18:31:48.613009 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.612955    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652"} err="failed to get container status \"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652\": rpc error: code = NotFound desc = could not find container \"1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652\": container with ID starting with 1b426a272935a99c0117306e97c65aa717c817930c28fc9b9f4bb2aa00472652 not found: ID does not exist"
Sep 11 18:31:48.617852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.617830    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:48.624834 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.624815    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1fc84f74-5bd6977565-6xkld"]
Sep 11 18:31:48.638913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.638895    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:48.642953 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:48.642936    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1fc84f74-5bcc568bf-h86s8"]
Sep 11 18:31:50.553714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:50.553678    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f099d93-d444-469c-a51f-dda2aea71ce2" path="/var/lib/kubelet/pods/0f099d93-d444-469c-a51f-dda2aea71ce2/volumes"
Sep 11 18:31:50.554070 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:50.554057    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="883955eb-dc65-4ad0-8a98-5fae25959e13" path="/var/lib/kubelet/pods/883955eb-dc65-4ad0-8a98-5fae25959e13/volumes"
Sep 11 18:31:50.554396 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:50.554381    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0aa6d43-c579-435e-90b2-76e68d1f721c" path="/var/lib/kubelet/pods/e0aa6d43-c579-435e-90b2-76e68d1f721c/volumes"
Sep 11 18:31:58.033480 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033410    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033751    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0aa6d43-c579-435e-90b2-76e68d1f721c" containerName="payload-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033762    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0aa6d43-c579-435e-90b2-76e68d1f721c" containerName="payload-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033783    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="883955eb-dc65-4ad0-8a98-5fae25959e13" containerName="maas-api"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033788    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="883955eb-dc65-4ad0-8a98-5fae25959e13" containerName="maas-api"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033797    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f099d93-d444-469c-a51f-dda2aea71ce2" containerName="payload-pre-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033802    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f099d93-d444-469c-a51f-dda2aea71ce2" containerName="payload-pre-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033857    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f099d93-d444-469c-a51f-dda2aea71ce2" containerName="payload-pre-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033864    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0aa6d43-c579-435e-90b2-76e68d1f721c" containerName="payload-processing"
Sep 11 18:31:58.033913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.033872    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="883955eb-dc65-4ad0-8a98-5fae25959e13" containerName="maas-api"
Sep 11 18:31:58.035938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.035923    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.038342 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.038320    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-f924630a-serving-cert\""
Sep 11 18:31:58.053110 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.053081    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:31:58.068824 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.068796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rd7\" (UniqueName: \"kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.068921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.068861    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.068921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.068906    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.069013 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.068935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.169888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.169861    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.170014 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.169893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.170014 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.169985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rd7\" (UniqueName: \"kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.170086 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.170028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.170620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.170599    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.172344 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.172323    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.172481 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.172459    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.181793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.181774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rd7\" (UniqueName: \"kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7\") pod \"maas-api-e2e-derive-f924630a-55d887d584-plvc6\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.346051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.345969    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:58.649813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.649732    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:31:58.652885 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:58.652859    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed0d153_947c_477c_9b0c_1b624c1c7b16.slice/crio-0d02775c79f17c6116bef4c0af932237f6c8c63e34f0f4d209595d4b672912a5 WatchSource:0}: Error finding container 0d02775c79f17c6116bef4c0af932237f6c8c63e34f0f4d209595d4b672912a5: Status 404 returned error can't find the container with id 0d02775c79f17c6116bef4c0af932237f6c8c63e34f0f4d209595d4b672912a5
Sep 11 18:31:58.866715 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.866680    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:31:58.870092 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.870073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:58.874937 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.874918    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f924630a\""
Sep 11 18:31:58.890247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.890217    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:31:58.942444 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.942419    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:31:58.945069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.945054    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:58.949517 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.949500    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f924630a-dockercfg-98czv\""
Sep 11 18:31:58.966141 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.966114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:31:58.978248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.976822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:58.978248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.976953    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5rs\" (UniqueName: \"kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:58.978248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.976992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28kpw\" (UniqueName: \"kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:58.978248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:58.977099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.077982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.077950    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.078345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.078008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.078345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.078056    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5rs\" (UniqueName: \"kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.078345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.078074    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28kpw\" (UniqueName: \"kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.078816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.078793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.078850 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.078793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.095841 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.095814    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5rs\" (UniqueName: \"kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs\") pod \"payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") " pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.098615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.098593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28kpw\" (UniqueName: \"kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw\") pod \"payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.181207 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.181185    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.254046 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.254009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.464228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.464204    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:31:59.465735 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:59.465710    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4918e86_2b4e_42b0_96b1_0ef8fcb007ee.slice/crio-e5f32769f9c622da783ea03c3dc6d8dd9ce5b08904acc52da313c956e5b3cdad WatchSource:0}: Error finding container e5f32769f9c622da783ea03c3dc6d8dd9ce5b08904acc52da313c956e5b3cdad: Status 404 returned error can't find the container with id e5f32769f9c622da783ea03c3dc6d8dd9ce5b08904acc52da313c956e5b3cdad
Sep 11 18:31:59.539120 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.539094    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:31:59.541654 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:31:59.541633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode609d453_cefd_45f7_bf69_0fcbc7d370ed.slice/crio-2092b00809166c9fb646af4f61bfcefca29d0888941a09533faaf4eb4048acd3 WatchSource:0}: Error finding container 2092b00809166c9fb646af4f61bfcefca29d0888941a09533faaf4eb4048acd3: Status 404 returned error can't find the container with id 2092b00809166c9fb646af4f61bfcefca29d0888941a09533faaf4eb4048acd3
Sep 11 18:31:59.649272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.649233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" event={"ID":"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee","Type":"ContainerStarted","Data":"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"}
Sep 11 18:31:59.649428 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.649380    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:31:59.649428 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.649408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" event={"ID":"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee","Type":"ContainerStarted","Data":"e5f32769f9c622da783ea03c3dc6d8dd9ce5b08904acc52da313c956e5b3cdad"}
Sep 11 18:31:59.650811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.650788    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" event={"ID":"e609d453-cefd-45f7-bf69-0fcbc7d370ed","Type":"ContainerStarted","Data":"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"}
Sep 11 18:31:59.650919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.650830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" event={"ID":"e609d453-cefd-45f7-bf69-0fcbc7d370ed","Type":"ContainerStarted","Data":"2092b00809166c9fb646af4f61bfcefca29d0888941a09533faaf4eb4048acd3"}
Sep 11 18:31:59.650919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.650858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:31:59.652391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.652367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" event={"ID":"eed0d153-947c-477c-9b0c-1b624c1c7b16","Type":"ContainerStarted","Data":"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"}
Sep 11 18:31:59.652489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.652403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" event={"ID":"eed0d153-947c-477c-9b0c-1b624c1c7b16","Type":"ContainerStarted","Data":"0d02775c79f17c6116bef4c0af932237f6c8c63e34f0f4d209595d4b672912a5"}
Sep 11 18:31:59.652537 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.652501    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:31:59.676890 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.676851    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" podStartSLOduration=1.67683914 podStartE2EDuration="1.67683914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:59.67555019 +0000 UTC m=+989.561522678" watchObservedRunningTime="2026-09-11 18:31:59.67683914 +0000 UTC m=+989.562811629"
Sep 11 18:31:59.703823 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.703773    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" podStartSLOduration=1.70375547 podStartE2EDuration="1.70375547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:59.701361119 +0000 UTC m=+989.587333607" watchObservedRunningTime="2026-09-11 18:31:59.70375547 +0000 UTC m=+989.589727961"
Sep 11 18:31:59.750366 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:31:59.750276    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" podStartSLOduration=1.249796388 podStartE2EDuration="1.750259443s" totalImagesPullingTime="500.463055ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:31:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:31:59.748207699 +0000 UTC m=+989.634180187" watchObservedRunningTime="2026-09-11 18:31:59.750259443 +0000 UTC m=+989.636231932"
Sep 11 18:32:05.663464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:05.663436    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:32:10.660350 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:10.660318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:32:10.660830 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:10.660458    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:32:34.964831 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:34.964797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:32:34.965221 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:34.965040    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" podUID="eed0d153-947c-477c-9b0c-1b624c1c7b16" containerName="maas-api" containerID="cri-o://94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3" gracePeriod=30
Sep 11 18:32:35.043722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.043685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:32:35.043985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.043956    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" podUID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" containerName="payload-processing" containerID="cri-o://5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f" gracePeriod=30
Sep 11 18:32:35.052232 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.052156    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:32:35.052425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.052388    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" podUID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" containerName="payload-pre-processing" containerID="cri-o://d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c" gracePeriod=30
Sep 11 18:32:35.319591 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.319502    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:32:35.337640 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.337616    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:32:35.352942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.352913    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:32:35.505032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.504948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca\") pod \"eed0d153-947c-477c-9b0c-1b624c1c7b16\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") "
Sep 11 18:32:35.505032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.504985    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume\") pod \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") "
Sep 11 18:32:35.505032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505015    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28kpw\" (UniqueName: \"kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw\") pod \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\" (UID: \"e609d453-cefd-45f7-bf69-0fcbc7d370ed\") "
Sep 11 18:32:35.505326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505072    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs\") pod \"eed0d153-947c-477c-9b0c-1b624c1c7b16\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") "
Sep 11 18:32:35.505326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505095    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume\") pod \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") "
Sep 11 18:32:35.505326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505116    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h5rs\" (UniqueName: \"kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs\") pod \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\" (UID: \"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee\") "
Sep 11 18:32:35.505326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505200    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls\") pod \"eed0d153-947c-477c-9b0c-1b624c1c7b16\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") "
Sep 11 18:32:35.505326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505286    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4rd7\" (UniqueName: \"kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7\") pod \"eed0d153-947c-477c-9b0c-1b624c1c7b16\" (UID: \"eed0d153-947c-477c-9b0c-1b624c1c7b16\") "
Sep 11 18:32:35.505558 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e609d453-cefd-45f7-bf69-0fcbc7d370ed" (UID: "e609d453-cefd-45f7-bf69-0fcbc7d370ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:35.505558 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505351    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "eed0d153-947c-477c-9b0c-1b624c1c7b16" (UID: "eed0d153-947c-477c-9b0c-1b624c1c7b16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:35.505719 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505603    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eed0d153-947c-477c-9b0c-1b624c1c7b16-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.505719 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505620    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e609d453-cefd-45f7-bf69-0fcbc7d370ed-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.505719 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.505619    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" (UID: "f4918e86-2b4e-42b0-96b1-0ef8fcb007ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:32:35.507315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.507292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eed0d153-947c-477c-9b0c-1b624c1c7b16" (UID: "eed0d153-947c-477c-9b0c-1b624c1c7b16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:32:35.507605 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.507592    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw" (OuterVolumeSpecName: "kube-api-access-28kpw") pod "e609d453-cefd-45f7-bf69-0fcbc7d370ed" (UID: "e609d453-cefd-45f7-bf69-0fcbc7d370ed"). InnerVolumeSpecName "kube-api-access-28kpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:35.507716 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.507701    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "eed0d153-947c-477c-9b0c-1b624c1c7b16" (UID: "eed0d153-947c-477c-9b0c-1b624c1c7b16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:32:35.507862 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.507840    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7" (OuterVolumeSpecName: "kube-api-access-g4rd7") pod "eed0d153-947c-477c-9b0c-1b624c1c7b16" (UID: "eed0d153-947c-477c-9b0c-1b624c1c7b16"). InnerVolumeSpecName "kube-api-access-g4rd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:35.507898 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.507871    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs" (OuterVolumeSpecName: "kube-api-access-5h5rs") pod "f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" (UID: "f4918e86-2b4e-42b0-96b1-0ef8fcb007ee"). InnerVolumeSpecName "kube-api-access-5h5rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:32:35.606916 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606881    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rd7\" (UniqueName: \"kubernetes.io/projected/eed0d153-947c-477c-9b0c-1b624c1c7b16-kube-api-access-g4rd7\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.606916 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606912    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28kpw\" (UniqueName: \"kubernetes.io/projected/e609d453-cefd-45f7-bf69-0fcbc7d370ed-kube-api-access-28kpw\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.606916 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606924    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.607147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606933    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.607147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606941    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5rs\" (UniqueName: \"kubernetes.io/projected/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee-kube-api-access-5h5rs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.607147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.606949    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eed0d153-947c-477c-9b0c-1b624c1c7b16-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:32:35.839732 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.839635    2582 generic.go:356] "Generic (PLEG): container finished" podID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" containerID="d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c" exitCode=0
Sep 11 18:32:35.839732 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.839697    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"
Sep 11 18:32:35.839961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.839728    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" event={"ID":"e609d453-cefd-45f7-bf69-0fcbc7d370ed","Type":"ContainerDied","Data":"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"}
Sep 11 18:32:35.839961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.839784    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk" event={"ID":"e609d453-cefd-45f7-bf69-0fcbc7d370ed","Type":"ContainerDied","Data":"2092b00809166c9fb646af4f61bfcefca29d0888941a09533faaf4eb4048acd3"}
Sep 11 18:32:35.839961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.839819    2582 scope.go:117] "RemoveContainer" containerID="d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"
Sep 11 18:32:35.841104 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.841083    2582 generic.go:356] "Generic (PLEG): container finished" podID="eed0d153-947c-477c-9b0c-1b624c1c7b16" containerID="94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3" exitCode=0
Sep 11 18:32:35.841219 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.841128    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"
Sep 11 18:32:35.841292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.841221    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" event={"ID":"eed0d153-947c-477c-9b0c-1b624c1c7b16","Type":"ContainerDied","Data":"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"}
Sep 11 18:32:35.841292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.841272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6" event={"ID":"eed0d153-947c-477c-9b0c-1b624c1c7b16","Type":"ContainerDied","Data":"0d02775c79f17c6116bef4c0af932237f6c8c63e34f0f4d209595d4b672912a5"}
Sep 11 18:32:35.842785 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.842759    2582 generic.go:356] "Generic (PLEG): container finished" podID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" containerID="5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f" exitCode=0
Sep 11 18:32:35.842899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.842793    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"
Sep 11 18:32:35.842899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.842833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" event={"ID":"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee","Type":"ContainerDied","Data":"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"}
Sep 11 18:32:35.842899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.842879    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm" event={"ID":"f4918e86-2b4e-42b0-96b1-0ef8fcb007ee","Type":"ContainerDied","Data":"e5f32769f9c622da783ea03c3dc6d8dd9ce5b08904acc52da313c956e5b3cdad"}
Sep 11 18:32:35.850718 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.850696    2582 scope.go:117] "RemoveContainer" containerID="d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"
Sep 11 18:32:35.851056 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:32:35.851032    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c\": container with ID starting with d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c not found: ID does not exist" containerID="d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"
Sep 11 18:32:35.851152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.851065    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c"} err="failed to get container status \"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c\": rpc error: code = NotFound desc = could not find container \"d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c\": container with ID starting with d2ed4661d8d1fb5733d294fc9c6d2ccef3fd0f178810680f634f07a80553180c not found: ID does not exist"
Sep 11 18:32:35.851152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.851087    2582 scope.go:117] "RemoveContainer" containerID="94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"
Sep 11 18:32:35.859423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.859405    2582 scope.go:117] "RemoveContainer" containerID="94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"
Sep 11 18:32:35.859702 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:32:35.859683    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3\": container with ID starting with 94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3 not found: ID does not exist" containerID="94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"
Sep 11 18:32:35.859805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.859710    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3"} err="failed to get container status \"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3\": rpc error: code = NotFound desc = could not find container \"94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3\": container with ID starting with 94d01fa86bd831a6788a2bd5a8970ae6bcea495640f81ac9772181c432b70fe3 not found: ID does not exist"
Sep 11 18:32:35.859805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.859727    2582 scope.go:117] "RemoveContainer" containerID="5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"
Sep 11 18:32:35.868020 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.867987    2582 scope.go:117] "RemoveContainer" containerID="5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"
Sep 11 18:32:35.868331 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:32:35.868310    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f\": container with ID starting with 5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f not found: ID does not exist" containerID="5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"
Sep 11 18:32:35.868409 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.868337    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f"} err="failed to get container status \"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f\": rpc error: code = NotFound desc = could not find container \"5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f\": container with ID starting with 5fb15a3080e32ecf25d3756c02918af65526c14efa69bc499f3d2b957e961a1f not found: ID does not exist"
Sep 11 18:32:35.869453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.869434    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:32:35.872603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.872582    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f924630a-6b5fb9cc94-l6twk"]
Sep 11 18:32:35.889037 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.888399    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:32:35.891573 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.891553    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-f924630a-55d887d584-plvc6"]
Sep 11 18:32:35.903995 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.903973    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:32:35.908667 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:35.908647    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f924630a-8664f5bc45-d5nwm"]
Sep 11 18:32:36.553691 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:36.553660    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" path="/var/lib/kubelet/pods/e609d453-cefd-45f7-bf69-0fcbc7d370ed/volumes"
Sep 11 18:32:36.554081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:36.554066    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed0d153-947c-477c-9b0c-1b624c1c7b16" path="/var/lib/kubelet/pods/eed0d153-947c-477c-9b0c-1b624c1c7b16/volumes"
Sep 11 18:32:36.554442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:32:36.554426    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" path="/var/lib/kubelet/pods/f4918e86-2b4e-42b0-96b1-0ef8fcb007ee/volumes"
Sep 11 18:33:02.876024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.875987    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"]
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876362    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" containerName="payload-processing"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876374    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" containerName="payload-processing"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876383    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" containerName="payload-pre-processing"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876389    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" containerName="payload-pre-processing"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876397    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed0d153-947c-477c-9b0c-1b624c1c7b16" containerName="maas-api"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876402    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed0d153-947c-477c-9b0c-1b624c1c7b16" containerName="maas-api"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876452    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed0d153-947c-477c-9b0c-1b624c1c7b16" containerName="maas-api"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876460    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4918e86-2b4e-42b0-96b1-0ef8fcb007ee" containerName="payload-processing"
Sep 11 18:33:02.876499 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.876466    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e609d453-cefd-45f7-bf69-0fcbc7d370ed" containerName="payload-pre-processing"
Sep 11 18:33:02.879103 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.879087    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:02.881641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.881624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:33:02.888646 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.888590    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"]
Sep 11 18:33:02.934773 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:02.934743    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp\" (UID: \"b00bbfb7-1fd0-4e45-a72f-277971788fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:03.035920 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.035890    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp\" (UID: \"b00bbfb7-1fd0-4e45-a72f-277971788fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:03.036565 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.036540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp\" (UID: \"b00bbfb7-1fd0-4e45-a72f-277971788fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:03.191185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.191137    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:03.459678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.459656    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"]
Sep 11 18:33:03.461814 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:33:03.461787    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00bbfb7_1fd0_4e45_a72f_277971788fd9.slice/crio-e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7 WatchSource:0}: Error finding container e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7: Status 404 returned error can't find the container with id e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7
Sep 11 18:33:03.986327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.986289    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp" event={"ID":"b00bbfb7-1fd0-4e45-a72f-277971788fd9","Type":"ContainerStarted","Data":"967f25fbed6eab8fb920781a59ef095f6f02c5fba2b04fb54d3d2046d631e6fe"}
Sep 11 18:33:03.986716 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:03.986356    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp" event={"ID":"b00bbfb7-1fd0-4e45-a72f-277971788fd9","Type":"ContainerStarted","Data":"e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7"}
Sep 11 18:33:04.002634 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:04.002586    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp" podStartSLOduration=2.002571319 podStartE2EDuration="2.002571319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:04.000311998 +0000 UTC m=+1053.886284488" watchObservedRunningTime="2026-09-11 18:33:04.002571319 +0000 UTC m=+1053.888543808"
Sep 11 18:33:04.994131 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:04.994093    2582 generic.go:356] "Generic (PLEG): container finished" podID="b00bbfb7-1fd0-4e45-a72f-277971788fd9" containerID="967f25fbed6eab8fb920781a59ef095f6f02c5fba2b04fb54d3d2046d631e6fe" exitCode=0
Sep 11 18:33:04.994501 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:04.994190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp" event={"ID":"b00bbfb7-1fd0-4e45-a72f-277971788fd9","Type":"ContainerDied","Data":"967f25fbed6eab8fb920781a59ef095f6f02c5fba2b04fb54d3d2046d631e6fe"}
Sep 11 18:33:06.138589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:06.138562    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:06.271538 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:06.271460    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca\") pod \"b00bbfb7-1fd0-4e45-a72f-277971788fd9\" (UID: \"b00bbfb7-1fd0-4e45-a72f-277971788fd9\") "
Sep 11 18:33:06.271771 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:06.271750    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b00bbfb7-1fd0-4e45-a72f-277971788fd9" (UID: "b00bbfb7-1fd0-4e45-a72f-277971788fd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:06.372460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:06.372426    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b00bbfb7-1fd0-4e45-a72f-277971788fd9-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:33:07.005373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:07.005337    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp" event={"ID":"b00bbfb7-1fd0-4e45-a72f-277971788fd9","Type":"ContainerDied","Data":"e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7"}
Sep 11 18:33:07.005373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:07.005370    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"
Sep 11 18:33:07.005586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:07.005394    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4c0fa84c9b77ff45822dc1c9ac121e8d019e96f45647c2162ee0276c5f26fd7"
Sep 11 18:33:13.512385 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.512353    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:33:13.512913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.512543    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" podUID="94d71c16-b624-4cee-a347-d33070c7a781" containerName="payload-pre-processing" containerID="cri-o://46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06" gracePeriod=30
Sep 11 18:33:13.675663 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.675631    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"]
Sep 11 18:33:13.678104 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.678077    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-71b1c2-6g4vp"]
Sep 11 18:33:13.760361 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.760336    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:33:13.841895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.841830    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume\") pod \"94d71c16-b624-4cee-a347-d33070c7a781\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") "
Sep 11 18:33:13.842029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.841937    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvrfl\" (UniqueName: \"kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl\") pod \"94d71c16-b624-4cee-a347-d33070c7a781\" (UID: \"94d71c16-b624-4cee-a347-d33070c7a781\") "
Sep 11 18:33:13.842243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.842220    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94d71c16-b624-4cee-a347-d33070c7a781" (UID: "94d71c16-b624-4cee-a347-d33070c7a781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:33:13.844033 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.844017    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl" (OuterVolumeSpecName: "kube-api-access-gvrfl") pod "94d71c16-b624-4cee-a347-d33070c7a781" (UID: "94d71c16-b624-4cee-a347-d33070c7a781"). InnerVolumeSpecName "kube-api-access-gvrfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:33:13.943649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.943624    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94d71c16-b624-4cee-a347-d33070c7a781-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:33:13.943649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:13.943647    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvrfl\" (UniqueName: \"kubernetes.io/projected/94d71c16-b624-4cee-a347-d33070c7a781-kube-api-access-gvrfl\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:33:14.039295 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.039266    2582 generic.go:356] "Generic (PLEG): container finished" podID="94d71c16-b624-4cee-a347-d33070c7a781" containerID="46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06" exitCode=0
Sep 11 18:33:14.039420 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.039320    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"
Sep 11 18:33:14.039420 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.039355    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" event={"ID":"94d71c16-b624-4cee-a347-d33070c7a781","Type":"ContainerDied","Data":"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"}
Sep 11 18:33:14.039420 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.039412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8" event={"ID":"94d71c16-b624-4cee-a347-d33070c7a781","Type":"ContainerDied","Data":"798a62d19cff8bb286f75733b974d14a956b350be03db379ce1743f743f3c767"}
Sep 11 18:33:14.039530 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.039442    2582 scope.go:117] "RemoveContainer" containerID="46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"
Sep 11 18:33:14.049325 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.049278    2582 scope.go:117] "RemoveContainer" containerID="46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"
Sep 11 18:33:14.049541 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:33:14.049522    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06\": container with ID starting with 46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06 not found: ID does not exist" containerID="46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"
Sep 11 18:33:14.049583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.049549    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06"} err="failed to get container status \"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06\": rpc error: code = NotFound desc = could not find container \"46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06\": container with ID starting with 46f7191e83ddff2be35c7443782faf3901a92a712e3528a711d6adcb9d611b06 not found: ID does not exist"
Sep 11 18:33:14.066541 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.066522    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:33:14.070473 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.070453    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-71b1c2-779c4c9b5d-w49w8"]
Sep 11 18:33:14.553360 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.553323    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94d71c16-b624-4cee-a347-d33070c7a781" path="/var/lib/kubelet/pods/94d71c16-b624-4cee-a347-d33070c7a781/volumes"
Sep 11 18:33:14.553835 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:14.553756    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b00bbfb7-1fd0-4e45-a72f-277971788fd9" path="/var/lib/kubelet/pods/b00bbfb7-1fd0-4e45-a72f-277971788fd9/volumes"
Sep 11 18:33:32.209114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209077    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:33:32.209485 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209473    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b00bbfb7-1fd0-4e45-a72f-277971788fd9" containerName="revoke-keys"
Sep 11 18:33:32.209529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209489    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b00bbfb7-1fd0-4e45-a72f-277971788fd9" containerName="revoke-keys"
Sep 11 18:33:32.209529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209522    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94d71c16-b624-4cee-a347-d33070c7a781" containerName="payload-pre-processing"
Sep 11 18:33:32.209620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209532    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="94d71c16-b624-4cee-a347-d33070c7a781" containerName="payload-pre-processing"
Sep 11 18:33:32.209620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209593    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="94d71c16-b624-4cee-a347-d33070c7a781" containerName="payload-pre-processing"
Sep 11 18:33:32.209620 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.209604    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b00bbfb7-1fd0-4e45-a72f-277971788fd9" containerName="revoke-keys"
Sep 11 18:33:32.214329 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.214309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.217337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.217311    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-54fc63-serving-cert\""
Sep 11 18:33:32.224566 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.224545    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:33:32.302502 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.302469    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.302656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.302508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzthc\" (UniqueName: \"kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.302656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.302550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.302656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.302582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.403196 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.403135    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.403381 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.403287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzthc\" (UniqueName: \"kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.403464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.403404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.403464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.403438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.404016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.403988    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.405537 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.405519    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.405743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.405725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.412213 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.412188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzthc\" (UniqueName: \"kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc\") pod \"maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.525597 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.525536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:32.796299 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.796224    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:33:32.799969 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:33:32.799938    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5409918_7345_47e2_997a_0cc85a20f95e.slice/crio-e5a948e70661c26eb22ffa83b628162b67f87ebb017b2d684abc5e969257cc0b WatchSource:0}: Error finding container e5a948e70661c26eb22ffa83b628162b67f87ebb017b2d684abc5e969257cc0b: Status 404 returned error can't find the container with id e5a948e70661c26eb22ffa83b628162b67f87ebb017b2d684abc5e969257cc0b
Sep 11 18:33:32.801357 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.801336    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:33:32.953445 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.953415    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:33:32.957675 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.957659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:32.960267 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.960240    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-54fc63\""
Sep 11 18:33:32.960397 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.960355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-54fc63-dockercfg-jrhlf\""
Sep 11 18:33:32.966516 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:32.966492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:33:33.016679 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.016647    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.016828 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.016685    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qksd\" (UniqueName: \"kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.118459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.118373    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qksd\" (UniqueName: \"kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.118614 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.118489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.119097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.119080    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.126501 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.126482    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qksd\" (UniqueName: \"kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd\") pod \"payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") " pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.137443 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.137416    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" event={"ID":"d5409918-7345-47e2-997a-0cc85a20f95e","Type":"ContainerStarted","Data":"e5a948e70661c26eb22ffa83b628162b67f87ebb017b2d684abc5e969257cc0b"}
Sep 11 18:33:33.270561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.270536    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:33.552128 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:33.552097    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:33:33.553919 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:33:33.553895    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20e4554_a3d7_45fc_8c07_5a49af8d63a9.slice/crio-ea50166fd3809d7b968dfccb8c70b73d35160f3ed8e8ce5db3aedcabaf9ad59f WatchSource:0}: Error finding container ea50166fd3809d7b968dfccb8c70b73d35160f3ed8e8ce5db3aedcabaf9ad59f: Status 404 returned error can't find the container with id ea50166fd3809d7b968dfccb8c70b73d35160f3ed8e8ce5db3aedcabaf9ad59f
Sep 11 18:33:34.143882 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.143844    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" event={"ID":"d20e4554-a3d7-45fc-8c07-5a49af8d63a9","Type":"ContainerStarted","Data":"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"}
Sep 11 18:33:34.144051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.143905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" event={"ID":"d20e4554-a3d7-45fc-8c07-5a49af8d63a9","Type":"ContainerStarted","Data":"ea50166fd3809d7b968dfccb8c70b73d35160f3ed8e8ce5db3aedcabaf9ad59f"}
Sep 11 18:33:34.144051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.143934    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:33:34.145561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.145537    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" event={"ID":"d5409918-7345-47e2-997a-0cc85a20f95e","Type":"ContainerStarted","Data":"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"}
Sep 11 18:33:34.145672 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.145589    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:34.160513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.160474    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" podStartSLOduration=2.160462316 podStartE2EDuration="2.160462316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:34.159436554 +0000 UTC m=+1084.045409047" watchObservedRunningTime="2026-09-11 18:33:34.160462316 +0000 UTC m=+1084.046434804"
Sep 11 18:33:34.177027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:34.176990    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" podStartSLOduration=1.746149055 podStartE2EDuration="2.176977183s" totalImagesPullingTime="430.828128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:33:34.17537642 +0000 UTC m=+1084.061348909" watchObservedRunningTime="2026-09-11 18:33:34.176977183 +0000 UTC m=+1084.062949672"
Sep 11 18:33:40.156054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:40.156024    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:33:45.152141 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:33:45.152100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:34:23.743673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.743640    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:34:23.747241 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.747225    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.750097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.750075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-489fe7a4-serving-cert\""
Sep 11 18:34:23.776549 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.776528    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:34:23.851321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.851286    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.851686 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.851658    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.851835 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.851820    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.851953 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.851929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8wrc\" (UniqueName: \"kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.953185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.953147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.953379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.953263    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.953379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.953288    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.953379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.953303    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8wrc\" (UniqueName: \"kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.953865 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.953838    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.955714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.955688    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.955813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.955752    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:23.961054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:23.961035    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8wrc\" (UniqueName: \"kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc\") pod \"maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:24.057576 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:24.057488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:24.338464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:24.338440    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:34:24.340470 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:34:24.340445    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a7e2e16_e249_4bd6_b0cc_418c2b1ae78a.slice/crio-f51480a356849f5278c7f911e32a8ad229f8469f010f5ac9e220e20ed1e62233 WatchSource:0}: Error finding container f51480a356849f5278c7f911e32a8ad229f8469f010f5ac9e220e20ed1e62233: Status 404 returned error can't find the container with id f51480a356849f5278c7f911e32a8ad229f8469f010f5ac9e220e20ed1e62233
Sep 11 18:34:24.399493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:24.399464    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" event={"ID":"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a","Type":"ContainerStarted","Data":"f51480a356849f5278c7f911e32a8ad229f8469f010f5ac9e220e20ed1e62233"}
Sep 11 18:34:25.406437 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:25.406394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" event={"ID":"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a","Type":"ContainerStarted","Data":"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"}
Sep 11 18:34:25.406883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:25.406511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:34:25.424856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:25.424806    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" podStartSLOduration=2.001021322 podStartE2EDuration="2.424792053s" totalImagesPullingTime="423.770731ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:34:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:34:25.422893418 +0000 UTC m=+1135.308865907" watchObservedRunningTime="2026-09-11 18:34:25.424792053 +0000 UTC m=+1135.310764541"
Sep 11 18:34:31.416929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:34:31.416902    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:35:07.813866 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:07.813827    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:35:07.814415 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:07.814099    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" podUID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" containerName="maas-api" containerID="cri-o://db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9" gracePeriod=30
Sep 11 18:35:08.059385 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.059360    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:35:08.160008 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.159980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8wrc\" (UniqueName: \"kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc\") pod \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") "
Sep 11 18:35:08.160237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.160026    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls\") pod \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") "
Sep 11 18:35:08.160237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.160042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs\") pod \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") "
Sep 11 18:35:08.160237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.160076    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca\") pod \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\" (UID: \"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a\") "
Sep 11 18:35:08.160514 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.160482    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" (UID: "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:08.162236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.162209    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" (UID: "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:08.162330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.162261    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" (UID: "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:08.162388 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.162374    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc" (OuterVolumeSpecName: "kube-api-access-w8wrc") pod "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" (UID: "6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a"). InnerVolumeSpecName "kube-api-access-w8wrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:08.261128 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.261091    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8wrc\" (UniqueName: \"kubernetes.io/projected/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-kube-api-access-w8wrc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.261128 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.261123    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.261128 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.261135    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.261393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.261147    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:08.623662 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.623625    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" containerID="db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9" exitCode=0
Sep 11 18:35:08.623834 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.623674    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"
Sep 11 18:35:08.623834 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.623715    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" event={"ID":"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a","Type":"ContainerDied","Data":"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"}
Sep 11 18:35:08.623834 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.623772    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp" event={"ID":"6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a","Type":"ContainerDied","Data":"f51480a356849f5278c7f911e32a8ad229f8469f010f5ac9e220e20ed1e62233"}
Sep 11 18:35:08.623834 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.623799    2582 scope.go:117] "RemoveContainer" containerID="db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"
Sep 11 18:35:08.632534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.632516    2582 scope.go:117] "RemoveContainer" containerID="db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"
Sep 11 18:35:08.632807 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:35:08.632789    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9\": container with ID starting with db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9 not found: ID does not exist" containerID="db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"
Sep 11 18:35:08.632856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.632815    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9"} err="failed to get container status \"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9\": rpc error: code = NotFound desc = could not find container \"db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9\": container with ID starting with db94eafa7119b2ff97d7a681a6aea68658613ccee0c8e685d9f61d93cdab2ea9 not found: ID does not exist"
Sep 11 18:35:08.640284 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.640258    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:35:08.644109 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:08.644090    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-489fe7a4-765cb87c9c-ppqsp"]
Sep 11 18:35:10.553862 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:10.553824    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" path="/var/lib/kubelet/pods/6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a/volumes"
Sep 11 18:35:17.109895 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.109859    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"]
Sep 11 18:35:17.110371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.110225    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" containerName="maas-api"
Sep 11 18:35:17.110371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.110238    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" containerName="maas-api"
Sep 11 18:35:17.110371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.110314    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a7e2e16-e249-4bd6-b0cc-418c2b1ae78a" containerName="maas-api"
Sep 11 18:35:17.114631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.114613    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:17.117012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.116989    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:35:17.121067 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.121043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"]
Sep 11 18:35:17.242031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.241988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-54fc63-b92fd\" (UID: \"75525d9e-7c76-4169-bf92-0e6932429fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:17.342666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.342630    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-54fc63-b92fd\" (UID: \"75525d9e-7c76-4169-bf92-0e6932429fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:17.343257 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.343237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-54fc63-b92fd\" (UID: \"75525d9e-7c76-4169-bf92-0e6932429fcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:17.427426 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.427394    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:17.697571 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:17.697548    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"]
Sep 11 18:35:17.700469 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:35:17.700437    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75525d9e_7c76_4169_bf92_0e6932429fcc.slice/crio-0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2 WatchSource:0}: Error finding container 0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2: Status 404 returned error can't find the container with id 0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2
Sep 11 18:35:18.675631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:18.675540    2582 generic.go:356] "Generic (PLEG): container finished" podID="75525d9e-7c76-4169-bf92-0e6932429fcc" containerID="8303aa80de4d74af3cbb35a9cbb439443373d13699668c664b7fab8542670a72" exitCode=0
Sep 11 18:35:18.676019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:18.675622    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd" event={"ID":"75525d9e-7c76-4169-bf92-0e6932429fcc","Type":"ContainerDied","Data":"8303aa80de4d74af3cbb35a9cbb439443373d13699668c664b7fab8542670a72"}
Sep 11 18:35:18.676019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:18.675672    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd" event={"ID":"75525d9e-7c76-4169-bf92-0e6932429fcc","Type":"ContainerStarted","Data":"0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2"}
Sep 11 18:35:19.817808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:19.817782    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:19.967101 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:19.967002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca\") pod \"75525d9e-7c76-4169-bf92-0e6932429fcc\" (UID: \"75525d9e-7c76-4169-bf92-0e6932429fcc\") "
Sep 11 18:35:19.967395 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:19.967372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75525d9e-7c76-4169-bf92-0e6932429fcc" (UID: "75525d9e-7c76-4169-bf92-0e6932429fcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:20.068202 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:20.068142    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75525d9e-7c76-4169-bf92-0e6932429fcc-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:20.686341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:20.686309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"
Sep 11 18:35:20.686341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:20.686328    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd" event={"ID":"75525d9e-7c76-4169-bf92-0e6932429fcc","Type":"ContainerDied","Data":"0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2"}
Sep 11 18:35:20.686598 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:20.686387    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f741cb423eb70f0a1ade388b02b9b1ca0d025ff7e02b89cce81b9d3fc1d9cc2"
Sep 11 18:35:27.678139 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.678103    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:35:27.678578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.678349    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" podUID="d5409918-7345-47e2-997a-0cc85a20f95e" containerName="maas-api" containerID="cri-o://26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31" gracePeriod=30
Sep 11 18:35:27.747657 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.747626    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:35:27.747854 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.747832    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" podUID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" containerName="payload-processing" containerID="cri-o://00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13" gracePeriod=30
Sep 11 18:35:27.941860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.941380    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"]
Sep 11 18:35:27.944233 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:27.944206    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-54fc63-b92fd"]
Sep 11 18:35:28.007031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.006983    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:35:28.038078 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.038046    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:35:28.149266 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149129    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs\") pod \"d5409918-7345-47e2-997a-0cc85a20f95e\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") "
Sep 11 18:35:28.149266 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149227    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume\") pod \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") "
Sep 11 18:35:28.149509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149311    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qksd\" (UniqueName: \"kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd\") pod \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\" (UID: \"d20e4554-a3d7-45fc-8c07-5a49af8d63a9\") "
Sep 11 18:35:28.149509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls\") pod \"d5409918-7345-47e2-997a-0cc85a20f95e\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") "
Sep 11 18:35:28.149509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149405    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzthc\" (UniqueName: \"kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc\") pod \"d5409918-7345-47e2-997a-0cc85a20f95e\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") "
Sep 11 18:35:28.149676 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149525    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca\") pod \"d5409918-7345-47e2-997a-0cc85a20f95e\" (UID: \"d5409918-7345-47e2-997a-0cc85a20f95e\") "
Sep 11 18:35:28.149676 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149536    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d20e4554-a3d7-45fc-8c07-5a49af8d63a9" (UID: "d20e4554-a3d7-45fc-8c07-5a49af8d63a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:28.149931 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.149908    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.150396 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.150355    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5409918-7345-47e2-997a-0cc85a20f95e" (UID: "d5409918-7345-47e2-997a-0cc85a20f95e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:35:28.151656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.151620    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d5409918-7345-47e2-997a-0cc85a20f95e" (UID: "d5409918-7345-47e2-997a-0cc85a20f95e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:28.152186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.152147    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5409918-7345-47e2-997a-0cc85a20f95e" (UID: "d5409918-7345-47e2-997a-0cc85a20f95e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:35:28.152796 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.152772    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd" (OuterVolumeSpecName: "kube-api-access-9qksd") pod "d20e4554-a3d7-45fc-8c07-5a49af8d63a9" (UID: "d20e4554-a3d7-45fc-8c07-5a49af8d63a9"). InnerVolumeSpecName "kube-api-access-9qksd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:28.152920 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.152895    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc" (OuterVolumeSpecName: "kube-api-access-wzthc") pod "d5409918-7345-47e2-997a-0cc85a20f95e" (UID: "d5409918-7345-47e2-997a-0cc85a20f95e"). InnerVolumeSpecName "kube-api-access-wzthc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:35:28.251383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.251303    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qksd\" (UniqueName: \"kubernetes.io/projected/d20e4554-a3d7-45fc-8c07-5a49af8d63a9-kube-api-access-9qksd\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.251383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.251332    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.251383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.251342    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzthc\" (UniqueName: \"kubernetes.io/projected/d5409918-7345-47e2-997a-0cc85a20f95e-kube-api-access-wzthc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.251383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.251350    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5409918-7345-47e2-997a-0cc85a20f95e-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.251383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.251358    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5409918-7345-47e2-997a-0cc85a20f95e-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:35:28.553678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.553599    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75525d9e-7c76-4169-bf92-0e6932429fcc" path="/var/lib/kubelet/pods/75525d9e-7c76-4169-bf92-0e6932429fcc/volumes"
Sep 11 18:35:28.728826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.728794    2582 generic.go:356] "Generic (PLEG): container finished" podID="d5409918-7345-47e2-997a-0cc85a20f95e" containerID="26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31" exitCode=0
Sep 11 18:35:28.729242 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.728841    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"
Sep 11 18:35:28.729242 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.728873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" event={"ID":"d5409918-7345-47e2-997a-0cc85a20f95e","Type":"ContainerDied","Data":"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"}
Sep 11 18:35:28.729242 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.728921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc" event={"ID":"d5409918-7345-47e2-997a-0cc85a20f95e","Type":"ContainerDied","Data":"e5a948e70661c26eb22ffa83b628162b67f87ebb017b2d684abc5e969257cc0b"}
Sep 11 18:35:28.729242 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.728943    2582 scope.go:117] "RemoveContainer" containerID="26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"
Sep 11 18:35:28.730327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.730250    2582 generic.go:356] "Generic (PLEG): container finished" podID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" containerID="00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13" exitCode=0
Sep 11 18:35:28.730327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.730295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" event={"ID":"d20e4554-a3d7-45fc-8c07-5a49af8d63a9","Type":"ContainerDied","Data":"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"}
Sep 11 18:35:28.730327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.730304    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"
Sep 11 18:35:28.730509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.730332    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd" event={"ID":"d20e4554-a3d7-45fc-8c07-5a49af8d63a9","Type":"ContainerDied","Data":"ea50166fd3809d7b968dfccb8c70b73d35160f3ed8e8ce5db3aedcabaf9ad59f"}
Sep 11 18:35:28.737743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.737726    2582 scope.go:117] "RemoveContainer" containerID="26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"
Sep 11 18:35:28.738025 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:35:28.737998    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31\": container with ID starting with 26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31 not found: ID does not exist" containerID="26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"
Sep 11 18:35:28.738122 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.738025    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31"} err="failed to get container status \"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31\": rpc error: code = NotFound desc = could not find container \"26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31\": container with ID starting with 26dda036b72e78dc05377fb790148f7c05a5d65f5cba5ffbf1cf6e16a449bf31 not found: ID does not exist"
Sep 11 18:35:28.738122 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.738042    2582 scope.go:117] "RemoveContainer" containerID="00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"
Sep 11 18:35:28.745557 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.745539    2582 scope.go:117] "RemoveContainer" containerID="00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"
Sep 11 18:35:28.745813 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:35:28.745792    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13\": container with ID starting with 00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13 not found: ID does not exist" containerID="00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"
Sep 11 18:35:28.745899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.745821    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13"} err="failed to get container status \"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13\": rpc error: code = NotFound desc = could not find container \"00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13\": container with ID starting with 00cca2c1ef91d913d2e732a796d42e17603daee97a481206b192f6e325963e13 not found: ID does not exist"
Sep 11 18:35:28.748766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.748745    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:35:28.753224 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.753204    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-54fc63-5d8b86f697-2jhcc"]
Sep 11 18:35:28.765465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.765439    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:35:28.768630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:28.768611    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-54fc63-c5fcc4b5-9mcgd"]
Sep 11 18:35:30.554413 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:30.554377    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" path="/var/lib/kubelet/pods/d20e4554-a3d7-45fc-8c07-5a49af8d63a9/volumes"
Sep 11 18:35:30.554903 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:30.554782    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5409918-7345-47e2-997a-0cc85a20f95e" path="/var/lib/kubelet/pods/d5409918-7345-47e2-997a-0cc85a20f95e/volumes"
Sep 11 18:35:41.258628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.258591    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:35:41.259186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259141    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5409918-7345-47e2-997a-0cc85a20f95e" containerName="maas-api"
Sep 11 18:35:41.259320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259192    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5409918-7345-47e2-997a-0cc85a20f95e" containerName="maas-api"
Sep 11 18:35:41.259320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259221    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" containerName="payload-processing"
Sep 11 18:35:41.259320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259234    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" containerName="payload-processing"
Sep 11 18:35:41.259320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259248    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75525d9e-7c76-4169-bf92-0e6932429fcc" containerName="revoke-keys"
Sep 11 18:35:41.259320 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259254    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="75525d9e-7c76-4169-bf92-0e6932429fcc" containerName="revoke-keys"
Sep 11 18:35:41.259495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259347    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d20e4554-a3d7-45fc-8c07-5a49af8d63a9" containerName="payload-processing"
Sep 11 18:35:41.259495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259356    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5409918-7345-47e2-997a-0cc85a20f95e" containerName="maas-api"
Sep 11 18:35:41.259495 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.259362    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="75525d9e-7c76-4169-bf92-0e6932429fcc" containerName="revoke-keys"
Sep 11 18:35:41.263492 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.263474    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.266030 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.266008    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-93d67f-serving-cert\""
Sep 11 18:35:41.266906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.266880    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.267004 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.266921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.267004 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.266949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.267117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.267021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tklmg\" (UniqueName: \"kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.284065 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.284028    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:35:41.367653 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.367618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.367653 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.367656    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.367899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.367679    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.367899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.367699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tklmg\" (UniqueName: \"kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.367899 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:35:41.367752    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-serving-cert: secret "maas-api-e2e-shared-a-w2-93d67f-serving-cert" not found
Sep 11 18:35:41.367899 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:35:41.367821    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls podName:451ca7f9-9cbd-4377-bff4-0801c1a75249 nodeName:}" failed. No retries permitted until 2026-09-11 18:35:41.867805008 +0000 UTC m=+1211.753777475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls") pod "maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" (UID: "451ca7f9-9cbd-4377-bff4-0801c1a75249") : secret "maas-api-e2e-shared-a-w2-93d67f-serving-cert" not found
Sep 11 18:35:41.368455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.368430    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.369974 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.369957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.376589 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.376567    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tklmg\" (UniqueName: \"kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.856814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.856769    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:35:41.860610 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.860585    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.863193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.863141    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-93d67f-dockercfg-z6tkq\""
Sep 11 18:35:41.863193 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.863163    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-93d67f\""
Sep 11 18:35:41.872747 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.872724    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.872899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.872786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.872899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.872842    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9zw\" (UniqueName: \"kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.875647 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.875623    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.877105 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.877080    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:35:41.878795 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.878772    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:35:41.883134 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.883115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:41.885276 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.885254    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:41.903281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.903257    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:35:41.975574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.974027    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht5j9\" (UniqueName: \"kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:41.975574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.974076    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:41.975574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.974115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9zw\" (UniqueName: \"kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.975574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.974247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.975574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.975209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:41.988712 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:41.988646    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9zw\" (UniqueName: \"kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw\") pod \"payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:42.075548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.075514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht5j9\" (UniqueName: \"kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.075548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.075551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.076287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.076267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.087330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.087307    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht5j9\" (UniqueName: \"kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9\") pod \"payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.171718 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.171690    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:42.176763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.176741    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:35:42.178375 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:35:42.178352    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451ca7f9_9cbd_4377_bff4_0801c1a75249.slice/crio-5731fd32f67350a6964edd41f77b0b136ea56f23fa66ad47e3c9cc52cfa411e0 WatchSource:0}: Error finding container 5731fd32f67350a6964edd41f77b0b136ea56f23fa66ad47e3c9cc52cfa411e0: Status 404 returned error can't find the container with id 5731fd32f67350a6964edd41f77b0b136ea56f23fa66ad47e3c9cc52cfa411e0
Sep 11 18:35:42.195676 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.195647    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.448956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.448931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:35:42.450670 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:35:42.450643    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc879beeb_e297_4e19_96c7_e5aa69255154.slice/crio-e7daad12cf0374c96b3949ab01d0a0c87cbb8fafb8ce26160218744569c95ada WatchSource:0}: Error finding container e7daad12cf0374c96b3949ab01d0a0c87cbb8fafb8ce26160218744569c95ada: Status 404 returned error can't find the container with id e7daad12cf0374c96b3949ab01d0a0c87cbb8fafb8ce26160218744569c95ada
Sep 11 18:35:42.474521 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.474439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:35:42.483394 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:35:42.483364    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad0abde1_71da_47c5_a6ad_ab5431864d86.slice/crio-e7682e876bf63fdda69f32de2d4e95fd239bd6d03ecf1fcd7543e2e05096fb52 WatchSource:0}: Error finding container e7682e876bf63fdda69f32de2d4e95fd239bd6d03ecf1fcd7543e2e05096fb52: Status 404 returned error can't find the container with id e7682e876bf63fdda69f32de2d4e95fd239bd6d03ecf1fcd7543e2e05096fb52
Sep 11 18:35:42.803317 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.803220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" event={"ID":"ad0abde1-71da-47c5-a6ad-ab5431864d86","Type":"ContainerStarted","Data":"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"}
Sep 11 18:35:42.803317 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.803288    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" event={"ID":"ad0abde1-71da-47c5-a6ad-ab5431864d86","Type":"ContainerStarted","Data":"e7682e876bf63fdda69f32de2d4e95fd239bd6d03ecf1fcd7543e2e05096fb52"}
Sep 11 18:35:42.803561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.803551    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:35:42.805092 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.805065    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" event={"ID":"c879beeb-e297-4e19-96c7-e5aa69255154","Type":"ContainerStarted","Data":"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"}
Sep 11 18:35:42.805237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.805128    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:42.805237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.805156    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" event={"ID":"c879beeb-e297-4e19-96c7-e5aa69255154","Type":"ContainerStarted","Data":"e7daad12cf0374c96b3949ab01d0a0c87cbb8fafb8ce26160218744569c95ada"}
Sep 11 18:35:42.806772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.806747    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" event={"ID":"451ca7f9-9cbd-4377-bff4-0801c1a75249","Type":"ContainerStarted","Data":"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"}
Sep 11 18:35:42.806898 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.806804    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" event={"ID":"451ca7f9-9cbd-4377-bff4-0801c1a75249","Type":"ContainerStarted","Data":"5731fd32f67350a6964edd41f77b0b136ea56f23fa66ad47e3c9cc52cfa411e0"}
Sep 11 18:35:42.806898 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.806832    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:42.821816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.821774    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" podStartSLOduration=1.8217603530000002 podStartE2EDuration="1.821760353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:42.820127166 +0000 UTC m=+1212.706099656" watchObservedRunningTime="2026-09-11 18:35:42.821760353 +0000 UTC m=+1212.707732841"
Sep 11 18:35:42.840121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.840075    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" podStartSLOduration=1.470308847 podStartE2EDuration="1.840061374s" totalImagesPullingTime="369.752527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:42.83885887 +0000 UTC m=+1212.724831372" watchObservedRunningTime="2026-09-11 18:35:42.840061374 +0000 UTC m=+1212.726033865"
Sep 11 18:35:42.856763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:42.856719    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" podStartSLOduration=1.856705292 podStartE2EDuration="1.856705292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:35:42.855349265 +0000 UTC m=+1212.741321776" watchObservedRunningTime="2026-09-11 18:35:42.856705292 +0000 UTC m=+1212.742677783"
Sep 11 18:35:48.817810 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:48.817780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:35:53.814476 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:53.814349    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:35:53.818821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:35:53.814579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:36:16.478572 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.478534    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:36:16.483322 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.483300    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.485909 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.485885    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-e6454f-dockercfg-f4mz9\""
Sep 11 18:36:16.486016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.485890    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-e6454f\""
Sep 11 18:36:16.494007 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.493980    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:36:16.525115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.525080    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:36:16.528658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.528637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.540559 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.540533    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:36:16.585764 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.585721    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.585989 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.585833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq4kq\" (UniqueName: \"kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.686433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.686382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.686636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.686444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.686636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.686518    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw9dk\" (UniqueName: \"kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.686758 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.686639    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq4kq\" (UniqueName: \"kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.687372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.687344    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.696252 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.696230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq4kq\" (UniqueName: \"kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq\") pod \"payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.787324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.787218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.787494 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.787328    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw9dk\" (UniqueName: \"kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.788236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.788209    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.795859 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.795829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw9dk\" (UniqueName: \"kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk\") pod \"payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:16.797500 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.797473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:16.838007 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:16.837978    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:17.083999 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.083972    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:36:17.085618 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:36:17.085596    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0d8351_e77f_4566_854b_30897f10f78f.slice/crio-b11f95d03950b4c0a987b7c4575a2690da6d0860ff164f2ea21d516bddde582a WatchSource:0}: Error finding container b11f95d03950b4c0a987b7c4575a2690da6d0860ff164f2ea21d516bddde582a: Status 404 returned error can't find the container with id b11f95d03950b4c0a987b7c4575a2690da6d0860ff164f2ea21d516bddde582a
Sep 11 18:36:17.134428 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.134400    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:36:17.136010 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:36:17.135987    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b6ce23_3bcb_4e59_a29e_8dc9c55ceb5e.slice/crio-a8c66720c39cf01fc2fc71fb3928be32684d4c047863a127b3372b4e2be12105 WatchSource:0}: Error finding container a8c66720c39cf01fc2fc71fb3928be32684d4c047863a127b3372b4e2be12105: Status 404 returned error can't find the container with id a8c66720c39cf01fc2fc71fb3928be32684d4c047863a127b3372b4e2be12105
Sep 11 18:36:17.992015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.991875    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" event={"ID":"3b0d8351-e77f-4566-854b-30897f10f78f","Type":"ContainerStarted","Data":"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"}
Sep 11 18:36:17.992015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.991944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" event={"ID":"3b0d8351-e77f-4566-854b-30897f10f78f","Type":"ContainerStarted","Data":"b11f95d03950b4c0a987b7c4575a2690da6d0860ff164f2ea21d516bddde582a"}
Sep 11 18:36:17.992015 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.991974    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:17.994081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.994051    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" event={"ID":"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e","Type":"ContainerStarted","Data":"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"}
Sep 11 18:36:17.994255 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.994105    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" event={"ID":"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e","Type":"ContainerStarted","Data":"a8c66720c39cf01fc2fc71fb3928be32684d4c047863a127b3372b4e2be12105"}
Sep 11 18:36:17.994255 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:17.994134    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:18.009642 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:18.009601    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" podStartSLOduration=2.009585845 podStartE2EDuration="2.009585845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:18.007596426 +0000 UTC m=+1247.893568919" watchObservedRunningTime="2026-09-11 18:36:18.009585845 +0000 UTC m=+1247.895558333"
Sep 11 18:36:18.026839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:18.026791    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" podStartSLOduration=2.026776591 podStartE2EDuration="2.026776591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:18.025649972 +0000 UTC m=+1247.911622465" watchObservedRunningTime="2026-09-11 18:36:18.026776591 +0000 UTC m=+1247.912749083"
Sep 11 18:36:29.002277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:29.002198    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:36:29.002792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:29.002332    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:36:42.747550 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.747505    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:36:42.751056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.751031    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.754040 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.754016    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aab3a7\"/\"noten-model-aab3a7-kserve-self-signed-certs\""
Sep 11 18:36:42.754935 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.754911    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aab3a7\"/\"kube-root-ca.crt\""
Sep 11 18:36:42.755036 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.755013    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-aab3a7\"/\"default-dockercfg-md5xt\""
Sep 11 18:36:42.755140 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.755117    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-aab3a7\"/\"openshift-service-ca.crt\""
Sep 11 18:36:42.772765 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.772743    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:36:42.829893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.829863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.830038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.829898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.830038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.829920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtm8q\" (UniqueName: \"kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.830038 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.830024    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.830150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.830048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.830150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.830066    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.930948 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.930908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.930948 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.930955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.930986    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtm8q\" (UniqueName: \"kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931376 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931229    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931566 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.931636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.931616    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.933733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.933718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.934118 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.934098    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:42.942440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:42.942416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtm8q\" (UniqueName: \"kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q\") pod \"noten-model-aab3a7-kserve-599699b5f9-wjpjw\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") " pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:43.060764 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:43.060660    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:43.344142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:43.344114    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:36:43.345200 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:36:43.345142    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ebfb071_fa07_4d4a_bf5f_d85d520054ae.slice/crio-571f3beef03987749b017e69cdfb7a9a2036c0df1bee1eead7aceae04bad421e WatchSource:0}: Error finding container 571f3beef03987749b017e69cdfb7a9a2036c0df1bee1eead7aceae04bad421e: Status 404 returned error can't find the container with id 571f3beef03987749b017e69cdfb7a9a2036c0df1bee1eead7aceae04bad421e
Sep 11 18:36:44.133688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:44.133635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" event={"ID":"2ebfb071-fa07-4d4a-bf5f-d85d520054ae","Type":"ContainerStarted","Data":"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"}
Sep 11 18:36:44.133688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:44.133695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" event={"ID":"2ebfb071-fa07-4d4a-bf5f-d85d520054ae","Type":"ContainerStarted","Data":"571f3beef03987749b017e69cdfb7a9a2036c0df1bee1eead7aceae04bad421e"}
Sep 11 18:36:44.157356 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:44.157300    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" podStartSLOduration=2.15728278 podStartE2EDuration="2.15728278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:36:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:36:44.154678737 +0000 UTC m=+1274.040651227" watchObservedRunningTime="2026-09-11 18:36:44.15728278 +0000 UTC m=+1274.043255285"
Sep 11 18:36:53.061688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:53.061648    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:53.062093 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:53.061701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:53.074505 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:53.074475    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:36:53.196799 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:36:53.196762    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:37:01.245289 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.245256    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:37:01.248695 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.248679    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.251637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.251610    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8d2ddf84\""
Sep 11 18:37:01.251753 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.251610    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8d2ddf84-dockercfg-7hnwz\""
Sep 11 18:37:01.261565 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.261544    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:37:01.265399 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.265375    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:37:01.265512 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.265476    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.276232 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.276210    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:37:01.404746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.404707    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8zs6\" (UniqueName: \"kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.404746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.404749    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.405000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.404777    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n84j\" (UniqueName: \"kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.405000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.404972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.506205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.506205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506136    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8zs6\" (UniqueName: \"kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.506205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.506205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506196    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n84j\" (UniqueName: \"kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.506959 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506932    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.507068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.506935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.514073 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.514047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n84j\" (UniqueName: \"kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j\") pod \"payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") " pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.514198 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.514127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8zs6\" (UniqueName: \"kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6\") pod \"payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.558812 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.558758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:01.576944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.576920    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:01.839870 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.839845    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:37:01.841774 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:37:01.841732    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f95060_102a_4580_bd80_cb4f5d898271.slice/crio-7ba87dc5993d9e1dd21c06cb8b8913fc8eff7ea44603195ad85c7680c88f9a4a WatchSource:0}: Error finding container 7ba87dc5993d9e1dd21c06cb8b8913fc8eff7ea44603195ad85c7680c88f9a4a: Status 404 returned error can't find the container with id 7ba87dc5993d9e1dd21c06cb8b8913fc8eff7ea44603195ad85c7680c88f9a4a
Sep 11 18:37:01.862450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:01.862428    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:37:02.232789 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.232748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" event={"ID":"a8088c5d-7241-421d-88e9-14a4ed3f7f43","Type":"ContainerStarted","Data":"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"}
Sep 11 18:37:02.233003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.232858    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" event={"ID":"a8088c5d-7241-421d-88e9-14a4ed3f7f43","Type":"ContainerStarted","Data":"e7c1850be5536128f3844d166f736e3ab122ea3f2a6a00a56e72709d8c4bf47f"}
Sep 11 18:37:02.233003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.232894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:02.234704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.234679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" event={"ID":"84f95060-102a-4580-bd80-cb4f5d898271","Type":"ContainerStarted","Data":"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"}
Sep 11 18:37:02.234822 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.234716    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" event={"ID":"84f95060-102a-4580-bd80-cb4f5d898271","Type":"ContainerStarted","Data":"7ba87dc5993d9e1dd21c06cb8b8913fc8eff7ea44603195ad85c7680c88f9a4a"}
Sep 11 18:37:02.234822 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.234735    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:02.252951 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.252907    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" podStartSLOduration=1.252894057 podStartE2EDuration="1.252894057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:02.252156905 +0000 UTC m=+1292.138129397" watchObservedRunningTime="2026-09-11 18:37:02.252894057 +0000 UTC m=+1292.138866575"
Sep 11 18:37:02.271501 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:02.271454    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" podStartSLOduration=1.27143853 podStartE2EDuration="1.27143853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:37:02.270503663 +0000 UTC m=+1292.156476167" watchObservedRunningTime="2026-09-11 18:37:02.27143853 +0000 UTC m=+1292.157411019"
Sep 11 18:37:13.241372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:13.241336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:37:13.241886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:13.241579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:37:19.573842 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:19.573803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:37:30.660318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:30.660286    2582 scope.go:117] "RemoveContainer" containerID="d634e22c39847c930b20662aa46f9ebdef3a7e49d88ede58331804986d60ca83"
Sep 11 18:37:34.593108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.593031    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" podUID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" containerName="main" containerID="cri-o://40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8" gracePeriod=45
Sep 11 18:37:34.840775 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.840751    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:37:34.925885 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.925862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.925897    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.925936    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926025    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926212 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926079    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home" (OuterVolumeSpecName: "home") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:34.926212 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926090    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gtm8q\" (UniqueName: \"kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926212 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926146    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs\") pod \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\" (UID: \"2ebfb071-fa07-4d4a-bf5f-d85d520054ae\") "
Sep 11 18:37:34.926383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926239    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:34.926383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:34.926492 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926473    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:34.926531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926490    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:34.926531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.926506    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:34.928222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.928199    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm" (OuterVolumeSpecName: "dshm") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:34.928321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.928264    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q" (OuterVolumeSpecName: "kube-api-access-gtm8q") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "kube-api-access-gtm8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:34.928321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:34.928272    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2ebfb071-fa07-4d4a-bf5f-d85d520054ae" (UID: "2ebfb071-fa07-4d4a-bf5f-d85d520054ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:37:35.027530 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.027505    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtm8q\" (UniqueName: \"kubernetes.io/projected/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-kube-api-access-gtm8q\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:35.027530 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.027530    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:35.027683 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.027540    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ebfb071-fa07-4d4a-bf5f-d85d520054ae-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:35.426121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.426087    2582 generic.go:356] "Generic (PLEG): container finished" podID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" containerID="40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8" exitCode=0
Sep 11 18:37:35.426308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.426128    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"
Sep 11 18:37:35.426308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.426180    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" event={"ID":"2ebfb071-fa07-4d4a-bf5f-d85d520054ae","Type":"ContainerDied","Data":"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"}
Sep 11 18:37:35.426308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.426220    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw" event={"ID":"2ebfb071-fa07-4d4a-bf5f-d85d520054ae","Type":"ContainerDied","Data":"571f3beef03987749b017e69cdfb7a9a2036c0df1bee1eead7aceae04bad421e"}
Sep 11 18:37:35.426308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.426235    2582 scope.go:117] "RemoveContainer" containerID="40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"
Sep 11 18:37:35.435804 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.435785    2582 scope.go:117] "RemoveContainer" containerID="40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"
Sep 11 18:37:35.436064 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:37:35.436048    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8\": container with ID starting with 40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8 not found: ID does not exist" containerID="40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"
Sep 11 18:37:35.436118 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.436073    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8"} err="failed to get container status \"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8\": rpc error: code = NotFound desc = could not find container \"40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8\": container with ID starting with 40792032dade21ff408ea5563a0f3492538559716c03a1b2973beb2c6b9abbc8 not found: ID does not exist"
Sep 11 18:37:35.449129 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.449107    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:37:35.453562 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:35.453543    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-aab3a7/noten-model-aab3a7-kserve-599699b5f9-wjpjw"]
Sep 11 18:37:36.554029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:36.553993    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" path="/var/lib/kubelet/pods/2ebfb071-fa07-4d4a-bf5f-d85d520054ae/volumes"
Sep 11 18:37:46.730511 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:46.730476    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:37:46.731029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:46.730763    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerName="payload-processing" containerID="cri-o://5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e" gracePeriod=30
Sep 11 18:37:47.009113 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.009061    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:37:47.147265 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.147232    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc9lh\" (UniqueName: \"kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh\") pod \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") "
Sep 11 18:37:47.147439 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.147427    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume\") pod \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\" (UID: \"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1\") "
Sep 11 18:37:47.147790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.147757    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" (UID: "14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:47.149273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.149255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh" (OuterVolumeSpecName: "kube-api-access-nc9lh") pod "14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" (UID: "14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1"). InnerVolumeSpecName "kube-api-access-nc9lh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:47.248528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.248434    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc9lh\" (UniqueName: \"kubernetes.io/projected/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-kube-api-access-nc9lh\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.248528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.248475    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:47.501678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.501578    2582 generic.go:356] "Generic (PLEG): container finished" podID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerID="5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e" exitCode=0
Sep 11 18:37:47.501678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.501629    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"
Sep 11 18:37:47.501879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.501629    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" event={"ID":"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1","Type":"ContainerDied","Data":"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"}
Sep 11 18:37:47.501879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.501766    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" event={"ID":"14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1","Type":"ContainerDied","Data":"1379b59e70e155eb042a96c1a4af00e859604274b1576c664f4cf38e73372dcc"}
Sep 11 18:37:47.501879 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.501805    2582 scope.go:117] "RemoveContainer" containerID="5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"
Sep 11 18:37:47.510874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.510851    2582 scope.go:117] "RemoveContainer" containerID="5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"
Sep 11 18:37:47.511147 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:37:47.511127    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e\": container with ID starting with 5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e not found: ID does not exist" containerID="5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"
Sep 11 18:37:47.511234 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.511156    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e"} err="failed to get container status \"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e\": rpc error: code = NotFound desc = could not find container \"5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e\": container with ID starting with 5a40c947fa6e63cd5b3aac814809f7079c3b0149794c7482bd013c9316f2093e not found: ID does not exist"
Sep 11 18:37:47.527510 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.527488    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:37:47.531418 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.531397    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf"]
Sep 11 18:37:47.953436 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:47.953398    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-52b585-66df7776c-6r2hf" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.34:9004: i/o timeout"
Sep 11 18:37:48.553969 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:48.553939    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" path="/var/lib/kubelet/pods/14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1/volumes"
Sep 11 18:37:53.924838 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:53.924754    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:37:53.925297 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:53.925020    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy" containerID="cri-o://c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f" gracePeriod=30
Sep 11 18:37:59.178086 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.178062    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:37:59.262662 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262582    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262662 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262677    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262714    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gld4g\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262740    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262796    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.262884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262856    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data\") pod \"d7481c80-a626-4b7d-a4f5-be4848f80c85\" (UID: \"d7481c80-a626-4b7d-a4f5-be4848f80c85\") "
Sep 11 18:37:59.263311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.262990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:59.263311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.263217    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:59.263311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.263228    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.263311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.263236    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:37:59.263311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.263240    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:59.263578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.263421    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:59.264946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.264923    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:59.264946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.264927    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:37:59.265076 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.264947    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g" (OuterVolumeSpecName: "kube-api-access-gld4g") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "kube-api-access-gld4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:37:59.265076 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.264991    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7481c80-a626-4b7d-a4f5-be4848f80c85" (UID: "d7481c80-a626-4b7d-a4f5-be4848f80c85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:37:59.363811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363772    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7481c80-a626-4b7d-a4f5-be4848f80c85-istiod-ca-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.363811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363805    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.363811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363815    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gld4g\" (UniqueName: \"kubernetes.io/projected/d7481c80-a626-4b7d-a4f5-be4848f80c85-kube-api-access-gld4g\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.364042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363825    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-workload-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.364042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363833    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-credential-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.364042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363842    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-data\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.364042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363851    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-podinfo\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.364042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.363858    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7481c80-a626-4b7d-a4f5-be4848f80c85-istio-envoy\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:37:59.568638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.568551    2582 generic.go:356] "Generic (PLEG): container finished" podID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerID="c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f" exitCode=0
Sep 11 18:37:59.568638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.568604    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"
Sep 11 18:37:59.568638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.568620    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerDied","Data":"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"}
Sep 11 18:37:59.568913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.568660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj" event={"ID":"d7481c80-a626-4b7d-a4f5-be4848f80c85","Type":"ContainerDied","Data":"8c48817ea417040fb228c15aadda22a7109d0b9ae063f9054e293778bc113ee0"}
Sep 11 18:37:59.568913 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.568682    2582 scope.go:117] "RemoveContainer" containerID="c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"
Sep 11 18:37:59.578450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.578429    2582 scope.go:117] "RemoveContainer" containerID="8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"
Sep 11 18:37:59.586370 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.586327    2582 scope.go:117] "RemoveContainer" containerID="c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"
Sep 11 18:37:59.586665 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:37:59.586646    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f\": container with ID starting with c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f not found: ID does not exist" containerID="c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"
Sep 11 18:37:59.586733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.586674    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f"} err="failed to get container status \"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f\": rpc error: code = NotFound desc = could not find container \"c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f\": container with ID starting with c69085f6e7d98c31522da989849eb2d7c7b1e44f132ef3038b42c91763e6f38f not found: ID does not exist"
Sep 11 18:37:59.586733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.586692    2582 scope.go:117] "RemoveContainer" containerID="8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"
Sep 11 18:37:59.586983 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:37:59.586965    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4\": container with ID starting with 8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4 not found: ID does not exist" containerID="8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"
Sep 11 18:37:59.587035 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.586988    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4"} err="failed to get container status \"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4\": rpc error: code = NotFound desc = could not find container \"8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4\": container with ID starting with 8fe8fda78f7322bf71d42df0114bd51b8e9ddeeba157505a010bf0aecd72a7a4 not found: ID does not exist"
Sep 11 18:37:59.590032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.590006    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:37:59.595639 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:37:59.595616    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-52b585-openshift-default-fbd66cf7b-8v4lj"]
Sep 11 18:38:00.554225 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:00.554192    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" path="/var/lib/kubelet/pods/d7481c80-a626-4b7d-a4f5-be4848f80c85/volumes"
Sep 11 18:38:11.984378 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:11.984344    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:38:11.984783 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:11.984551    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" containerID="cri-o://b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f" gracePeriod=30
Sep 11 18:38:16.549469 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:16.549421    2582 patch_prober.go:28] interesting pod/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:38:16.549924 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:16.549530    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:38:17.244027 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.244006    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:38:17.325487 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325416    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325487 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrbjk\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325487 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325489    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325505    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325552    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325602    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325680    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325689    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data" (OuterVolumeSpecName: "istio-data") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:38:17.325759 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325704    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket\") pod \"b46d2127-3530-4de8-9f3e-761834ebde59\" (UID: \"b46d2127-3530-4de8-9f3e-761834ebde59\") "
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325869    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.325992    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-data\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.326011    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.326067    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:38:17.326119 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.326093    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:38:17.327816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.327791    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token" (OuterVolumeSpecName: "istio-token") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:17.327921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.327810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk" (OuterVolumeSpecName: "kube-api-access-hrbjk") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "kube-api-access-hrbjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:38:17.327921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.327888    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:38:17.327921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.327888    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b46d2127-3530-4de8-9f3e-761834ebde59" (UID: "b46d2127-3530-4de8-9f3e-761834ebde59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:38:17.426631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426601    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b46d2127-3530-4de8-9f3e-761834ebde59-istiod-ca-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426628    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426631 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426636    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b46d2127-3530-4de8-9f3e-761834ebde59-istio-podinfo\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426645    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-credential-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426654    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrbjk\" (UniqueName: \"kubernetes.io/projected/b46d2127-3530-4de8-9f3e-761834ebde59-kube-api-access-hrbjk\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426663    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-istio-envoy\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.426860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.426671    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b46d2127-3530-4de8-9f3e-761834ebde59-workload-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:38:17.663042 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.663012    2582 generic.go:356] "Generic (PLEG): container finished" podID="b46d2127-3530-4de8-9f3e-761834ebde59" containerID="b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f" exitCode=0
Sep 11 18:38:17.663486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.663064    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"
Sep 11 18:38:17.663486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.663092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerDied","Data":"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"}
Sep 11 18:38:17.663486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.663132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk" event={"ID":"b46d2127-3530-4de8-9f3e-761834ebde59","Type":"ContainerDied","Data":"91d008ad9dca450ebb7331ffe9c347d402819253596e61b6fff8b56e79ce3d64"}
Sep 11 18:38:17.663486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.663153    2582 scope.go:117] "RemoveContainer" containerID="b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"
Sep 11 18:38:17.674998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.674970    2582 scope.go:117] "RemoveContainer" containerID="2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"
Sep 11 18:38:17.682950 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.682930    2582 scope.go:117] "RemoveContainer" containerID="b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"
Sep 11 18:38:17.683217 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:38:17.683194    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f\": container with ID starting with b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f not found: ID does not exist" containerID="b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"
Sep 11 18:38:17.683307 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.683224    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f"} err="failed to get container status \"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f\": rpc error: code = NotFound desc = could not find container \"b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f\": container with ID starting with b9183ffd8c5f95efd5f1058d25e42136ce116ae82d2950e11231f370fae9df2f not found: ID does not exist"
Sep 11 18:38:17.683307 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.683241    2582 scope.go:117] "RemoveContainer" containerID="2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"
Sep 11 18:38:17.683590 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:38:17.683571    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19\": container with ID starting with 2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19 not found: ID does not exist" containerID="2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"
Sep 11 18:38:17.683680 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.683597    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19"} err="failed to get container status \"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19\": rpc error: code = NotFound desc = could not find container \"2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19\": container with ID starting with 2a9aa298b75671cf1d0697b58c8086d357da66496b25660b8fb78ab9f826fc19 not found: ID does not exist"
Sep 11 18:38:17.685721 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.685701    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:38:17.696032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:17.696007    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d18cc-openshift-default-55494b4f4b-cdzkk"]
Sep 11 18:38:18.555051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:18.555019    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" path="/var/lib/kubelet/pods/b46d2127-3530-4de8-9f3e-761834ebde59/volumes"
Sep 11 18:38:21.226795 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.226756    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:38:21.227215 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227164    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerName="payload-processing"
Sep 11 18:38:21.227215 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227198    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerName="payload-processing"
Sep 11 18:38:21.227215 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227209    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.227215 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227215    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227233    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227240    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227249    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" containerName="main"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227253    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" containerName="main"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227314    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ebfb071-fa07-4d4a-bf5f-d85d520054ae" containerName="main"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227324    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="14fb1f08-8a3a-4ec5-b6e2-009ef20c28d1" containerName="payload-processing"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227330    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227335    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227341    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.227373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.227346    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.231845 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.231826    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.234115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.234090    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-ec9107-dockercfg-j5hg4\""
Sep 11 18:38:21.234227 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.234134    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-ec9107\""
Sep 11 18:38:21.241757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.241731    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:38:21.252243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.252220    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:38:21.252557 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.252545    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.252608 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.252558    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7481c80-a626-4b7d-a4f5-be4848f80c85" containerName="istio-proxy"
Sep 11 18:38:21.252608 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.252576    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.252608 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.252582    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46d2127-3530-4de8-9f3e-761834ebde59" containerName="istio-proxy"
Sep 11 18:38:21.255665 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.255640    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.270724 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.270701    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:38:21.360229 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.360191    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsm7x\" (UniqueName: \"kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.360422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.360265    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.360422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.360296    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.360422 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.360327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcwv\" (UniqueName: \"kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.461515 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.461482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsm7x\" (UniqueName: \"kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.461681 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.461550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.461681 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.461584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.461681 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.461615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcwv\" (UniqueName: \"kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.462302 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.462284    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.462394 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.462377    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.470334 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.470311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcwv\" (UniqueName: \"kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv\") pod \"payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.470440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.470371    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsm7x\" (UniqueName: \"kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x\") pod \"payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.543363 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.543298    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:21.566285 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.566265    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:21.817964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.817938    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:38:21.819766 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:38:21.819739    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459b04ed_a9d6_4390_9155_dd7672f29fc4.slice/crio-8567cd49b884b59b8fe975c62db38398e527c66c3eb4d384ce8f2197bfce6962 WatchSource:0}: Error finding container 8567cd49b884b59b8fe975c62db38398e527c66c3eb4d384ce8f2197bfce6962: Status 404 returned error can't find the container with id 8567cd49b884b59b8fe975c62db38398e527c66c3eb4d384ce8f2197bfce6962
Sep 11 18:38:21.847348 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:21.847328    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:38:21.851150 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:38:21.851125    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cafc7f3_e44b_4c26_a9dc_34faa9302317.slice/crio-ceef95206652d23987197ee2496a2b8dde2c08269b29d33e172b194a0accad7c WatchSource:0}: Error finding container ceef95206652d23987197ee2496a2b8dde2c08269b29d33e172b194a0accad7c: Status 404 returned error can't find the container with id ceef95206652d23987197ee2496a2b8dde2c08269b29d33e172b194a0accad7c
Sep 11 18:38:22.692018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.691979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" event={"ID":"8cafc7f3-e44b-4c26-a9dc-34faa9302317","Type":"ContainerStarted","Data":"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"}
Sep 11 18:38:22.692512 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.692077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" event={"ID":"8cafc7f3-e44b-4c26-a9dc-34faa9302317","Type":"ContainerStarted","Data":"ceef95206652d23987197ee2496a2b8dde2c08269b29d33e172b194a0accad7c"}
Sep 11 18:38:22.692769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.692749    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:22.694297 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.694274    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" event={"ID":"459b04ed-a9d6-4390-9155-dd7672f29fc4","Type":"ContainerStarted","Data":"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"}
Sep 11 18:38:22.694425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.694316    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" event={"ID":"459b04ed-a9d6-4390-9155-dd7672f29fc4","Type":"ContainerStarted","Data":"8567cd49b884b59b8fe975c62db38398e527c66c3eb4d384ce8f2197bfce6962"}
Sep 11 18:38:22.694425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.694347    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:22.710967 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.710906    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" podStartSLOduration=1.7108870409999999 podStartE2EDuration="1.710887041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:22.710034278 +0000 UTC m=+1372.596006777" watchObservedRunningTime="2026-09-11 18:38:22.710887041 +0000 UTC m=+1372.596859531"
Sep 11 18:38:22.731586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:22.731538    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" podStartSLOduration=1.731518932 podStartE2EDuration="1.731518932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:22.728691879 +0000 UTC m=+1372.614664371" watchObservedRunningTime="2026-09-11 18:38:22.731518932 +0000 UTC m=+1372.617491426"
Sep 11 18:38:33.701047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:33.701017    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:38:33.701468 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:33.701210    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:38:55.527032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.526999    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:38:55.530578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.530560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.533453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.533429    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-d6c4cc-serving-cert\""
Sep 11 18:38:55.544892 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.544866    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:38:55.572963 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.572932    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.573144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.572969    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.573144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.572994    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gqnf\" (UniqueName: \"kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.573144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.573086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.673883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.673847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.673883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.673886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.674149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.673919    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gqnf\" (UniqueName: \"kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.674149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.673955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.674578 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.674553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.676602 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.676575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.676706 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.676575    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.682718 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.682689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gqnf\" (UniqueName: \"kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf\") pod \"maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:55.841420 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:55.841323    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:56.124304 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.124279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:38:56.127733 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:38:56.127705    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb86bcea_fe3f_46fc_bf8b_8cafe4c175dc.slice/crio-7468ca50f7473814f59f9a8505140be404105803f9777c775b06dbbeb846d73d WatchSource:0}: Error finding container 7468ca50f7473814f59f9a8505140be404105803f9777c775b06dbbeb846d73d: Status 404 returned error can't find the container with id 7468ca50f7473814f59f9a8505140be404105803f9777c775b06dbbeb846d73d
Sep 11 18:38:56.129223 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.129202    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:38:56.876339 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.876248    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" event={"ID":"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc","Type":"ContainerStarted","Data":"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"}
Sep 11 18:38:56.876339 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.876301    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" event={"ID":"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc","Type":"ContainerStarted","Data":"7468ca50f7473814f59f9a8505140be404105803f9777c775b06dbbeb846d73d"}
Sep 11 18:38:56.876757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.876403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:38:56.899628 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:38:56.899578    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" podStartSLOduration=1.609003792 podStartE2EDuration="1.899559949s" totalImagesPullingTime="290.556157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:38:56.897544097 +0000 UTC m=+1406.783516587" watchObservedRunningTime="2026-09-11 18:38:56.899559949 +0000 UTC m=+1406.785532437"
Sep 11 18:39:02.889024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:02.888989    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:39:30.704954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:30.704886    2582 scope.go:117] "RemoveContainer" containerID="967f25fbed6eab8fb920781a59ef095f6f02c5fba2b04fb54d3d2046d631e6fe"
Sep 11 18:39:36.050969 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.050931    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:39:36.054642 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.054622    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.057977 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.057953    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"openshift-service-ca.crt\""
Sep 11 18:39:36.057977 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.057964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"kube-root-ca.crt\""
Sep 11 18:39:36.058152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.057954    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"auth-test-model-ec9107-kserve-self-signed-certs\""
Sep 11 18:39:36.058243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.058230    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-ec9107\"/\"default-dockercfg-59fq9\""
Sep 11 18:39:36.064159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.064139    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:39:36.148919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.148887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.148919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.148923    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.149116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.149000    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.149116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.149025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.149116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.149041    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmzxs\" (UniqueName: \"kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.149116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.149073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250457 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250457 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250505    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250533    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmzxs\" (UniqueName: \"kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250583    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250881    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.250996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.250970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.252906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.252886    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.253076 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.253060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.258383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.258362    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmzxs\" (UniqueName: \"kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs\") pod \"auth-test-model-ec9107-kserve-78f7f4968c-rwms7\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") " pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.364733 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.364651    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:36.636099 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:36.636034    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:39:36.638121 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:39:36.638092    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3384e3d_6987_4010_839b_50209fe1b383.slice/crio-3bf60a3828191475db5308c2f2c725b40e9724c027c93f5a2036314fdcaf02ec WatchSource:0}: Error finding container 3bf60a3828191475db5308c2f2c725b40e9724c027c93f5a2036314fdcaf02ec: Status 404 returned error can't find the container with id 3bf60a3828191475db5308c2f2c725b40e9724c027c93f5a2036314fdcaf02ec
Sep 11 18:39:37.082627 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:37.082593    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" event={"ID":"a3384e3d-6987-4010-839b-50209fe1b383","Type":"ContainerStarted","Data":"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"}
Sep 11 18:39:37.082992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:37.082644    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" event={"ID":"a3384e3d-6987-4010-839b-50209fe1b383","Type":"ContainerStarted","Data":"3bf60a3828191475db5308c2f2c725b40e9724c027c93f5a2036314fdcaf02ec"}
Sep 11 18:39:37.107433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:37.107366    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" podStartSLOduration=1.107341531 podStartE2EDuration="1.107341531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:39:37.098399219 +0000 UTC m=+1446.984371706" watchObservedRunningTime="2026-09-11 18:39:37.107341531 +0000 UTC m=+1446.993314023"
Sep 11 18:39:46.365371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:46.365339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:46.365836 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:46.365651    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:46.378292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:46.378264    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:47.148536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:47.148509    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:39:54.858527 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:54.858491    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:39:54.858915 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:54.858699    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" podUID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" containerName="payload-processing" containerID="cri-o://0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c" gracePeriod=30
Sep 11 18:39:54.871139 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:54.871112    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:39:54.871407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:54.871382    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" podUID="84f95060-102a-4580-bd80-cb4f5d898271" containerName="payload-pre-processing" containerID="cri-o://1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438" gracePeriod=30
Sep 11 18:39:55.160447 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.160421    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:39:55.162982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.162965    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:39:55.183902 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.183874    2582 generic.go:356] "Generic (PLEG): container finished" podID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" containerID="0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c" exitCode=0
Sep 11 18:39:55.184068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.183909    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"
Sep 11 18:39:55.184068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.183979    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" event={"ID":"a8088c5d-7241-421d-88e9-14a4ed3f7f43","Type":"ContainerDied","Data":"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"}
Sep 11 18:39:55.184068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.184039    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt" event={"ID":"a8088c5d-7241-421d-88e9-14a4ed3f7f43","Type":"ContainerDied","Data":"e7c1850be5536128f3844d166f736e3ab122ea3f2a6a00a56e72709d8c4bf47f"}
Sep 11 18:39:55.184068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.184069    2582 scope.go:117] "RemoveContainer" containerID="0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"
Sep 11 18:39:55.185796 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.185770    2582 generic.go:356] "Generic (PLEG): container finished" podID="84f95060-102a-4580-bd80-cb4f5d898271" containerID="1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438" exitCode=0
Sep 11 18:39:55.185886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.185815    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"
Sep 11 18:39:55.185886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.185830    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" event={"ID":"84f95060-102a-4580-bd80-cb4f5d898271","Type":"ContainerDied","Data":"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"}
Sep 11 18:39:55.185886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.185871    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5" event={"ID":"84f95060-102a-4580-bd80-cb4f5d898271","Type":"ContainerDied","Data":"7ba87dc5993d9e1dd21c06cb8b8913fc8eff7ea44603195ad85c7680c88f9a4a"}
Sep 11 18:39:55.201386 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.201333    2582 scope.go:117] "RemoveContainer" containerID="0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"
Sep 11 18:39:55.201730 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:39:55.201695    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c\": container with ID starting with 0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c not found: ID does not exist" containerID="0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"
Sep 11 18:39:55.202136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.201734    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c"} err="failed to get container status \"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c\": rpc error: code = NotFound desc = could not find container \"0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c\": container with ID starting with 0c92fe47e59ca94428577d2eabaffd66d4e322611a518c962e6e4bd416a8b24c not found: ID does not exist"
Sep 11 18:39:55.202136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.201757    2582 scope.go:117] "RemoveContainer" containerID="1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"
Sep 11 18:39:55.210447 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.210424    2582 scope.go:117] "RemoveContainer" containerID="1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"
Sep 11 18:39:55.210719 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:39:55.210698    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438\": container with ID starting with 1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438 not found: ID does not exist" containerID="1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"
Sep 11 18:39:55.210786 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.210739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438"} err="failed to get container status \"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438\": rpc error: code = NotFound desc = could not find container \"1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438\": container with ID starting with 1dd904ed5a4cb99eea78a959be585e0a6ca1c95ca7e876826a3e124caf086438 not found: ID does not exist"
Sep 11 18:39:55.331327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331293    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4n84j\" (UniqueName: \"kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j\") pod \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") "
Sep 11 18:39:55.331489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331368    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8zs6\" (UniqueName: \"kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6\") pod \"84f95060-102a-4580-bd80-cb4f5d898271\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") "
Sep 11 18:39:55.331489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331385    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume\") pod \"84f95060-102a-4580-bd80-cb4f5d898271\" (UID: \"84f95060-102a-4580-bd80-cb4f5d898271\") "
Sep 11 18:39:55.331489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331409    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume\") pod \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\" (UID: \"a8088c5d-7241-421d-88e9-14a4ed3f7f43\") "
Sep 11 18:39:55.331827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331801    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "84f95060-102a-4580-bd80-cb4f5d898271" (UID: "84f95060-102a-4580-bd80-cb4f5d898271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:55.331907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.331810    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8088c5d-7241-421d-88e9-14a4ed3f7f43" (UID: "a8088c5d-7241-421d-88e9-14a4ed3f7f43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:39:55.333414 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.333387    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j" (OuterVolumeSpecName: "kube-api-access-4n84j") pod "a8088c5d-7241-421d-88e9-14a4ed3f7f43" (UID: "a8088c5d-7241-421d-88e9-14a4ed3f7f43"). InnerVolumeSpecName "kube-api-access-4n84j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:55.333532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.333510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6" (OuterVolumeSpecName: "kube-api-access-h8zs6") pod "84f95060-102a-4580-bd80-cb4f5d898271" (UID: "84f95060-102a-4580-bd80-cb4f5d898271"). InnerVolumeSpecName "kube-api-access-h8zs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:39:55.432311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.432281    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8zs6\" (UniqueName: \"kubernetes.io/projected/84f95060-102a-4580-bd80-cb4f5d898271-kube-api-access-h8zs6\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.432311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.432306    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84f95060-102a-4580-bd80-cb4f5d898271-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.432311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.432315    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8088c5d-7241-421d-88e9-14a4ed3f7f43-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.432557 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.432325    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n84j\" (UniqueName: \"kubernetes.io/projected/a8088c5d-7241-421d-88e9-14a4ed3f7f43-kube-api-access-4n84j\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:39:55.508070 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.508045    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:39:55.512609 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.512586    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8d2ddf84-857f47bb89-klgrt"]
Sep 11 18:39:55.525361 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.525334    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:39:55.528967 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:55.528948    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8d2ddf84-867cc77f8-4xhr5"]
Sep 11 18:39:56.554185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:56.554131    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84f95060-102a-4580-bd80-cb4f5d898271" path="/var/lib/kubelet/pods/84f95060-102a-4580-bd80-cb4f5d898271/volumes"
Sep 11 18:39:56.554550 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:39:56.554526    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" path="/var/lib/kubelet/pods/a8088c5d-7241-421d-88e9-14a4ed3f7f43/volumes"
Sep 11 18:41:30.741220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:41:30.741186    2582 scope.go:117] "RemoveContainer" containerID="8303aa80de4d74af3cbb35a9cbb439443373d13699668c664b7fab8542670a72"
Sep 11 18:42:04.303090 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303053    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:42:04.303825 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303800    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" containerName="payload-processing"
Sep 11 18:42:04.303938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303828    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" containerName="payload-processing"
Sep 11 18:42:04.303938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303846    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84f95060-102a-4580-bd80-cb4f5d898271" containerName="payload-pre-processing"
Sep 11 18:42:04.303938 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303857    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="84f95060-102a-4580-bd80-cb4f5d898271" containerName="payload-pre-processing"
Sep 11 18:42:04.304060 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303959    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="84f95060-102a-4580-bd80-cb4f5d898271" containerName="payload-pre-processing"
Sep 11 18:42:04.304060 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.303978    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8088c5d-7241-421d-88e9-14a4ed3f7f43" containerName="payload-processing"
Sep 11 18:42:04.307407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.307385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.309761 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.309734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-6c4c23-serving-cert\""
Sep 11 18:42:04.316161 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.316137    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:42:04.375671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.375649    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.375807 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.375693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.375807 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.375713    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrvj9\" (UniqueName: \"kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.375807 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.375796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.477117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.477084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.477312 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.477191    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.477312 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.477225    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.477312 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.477244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrvj9\" (UniqueName: \"kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.477424 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:42:04.477332    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-serving-cert: secret "maas-api-e2e-api-a-6c4c23-serving-cert" not found
Sep 11 18:42:04.477424 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:42:04.477405    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls podName:41eb2eac-75fc-450f-b23e-1248fd5dffd1 nodeName:}" failed. No retries permitted until 2026-09-11 18:42:04.977388329 +0000 UTC m=+1594.863360813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls") pod "maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" (UID: "41eb2eac-75fc-450f-b23e-1248fd5dffd1") : secret "maas-api-e2e-api-a-6c4c23-serving-cert" not found
Sep 11 18:42:04.477891 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.477865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.479581 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.479565    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.488980 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.488957    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrvj9\" (UniqueName: \"kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.982156 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.982115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:04.984503 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:04.984479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") pod \"maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:05.221185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:05.221146    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:05.497682 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:05.497659    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:42:05.499361 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:42:05.499335    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41eb2eac_75fc_450f_b23e_1248fd5dffd1.slice/crio-77da6d1aa31eec524631fa9cdc7c41b6feab1e2a2cc40d8b9df1dce0d85ad286 WatchSource:0}: Error finding container 77da6d1aa31eec524631fa9cdc7c41b6feab1e2a2cc40d8b9df1dce0d85ad286: Status 404 returned error can't find the container with id 77da6d1aa31eec524631fa9cdc7c41b6feab1e2a2cc40d8b9df1dce0d85ad286
Sep 11 18:42:05.846959 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:05.846929    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" event={"ID":"41eb2eac-75fc-450f-b23e-1248fd5dffd1","Type":"ContainerStarted","Data":"77da6d1aa31eec524631fa9cdc7c41b6feab1e2a2cc40d8b9df1dce0d85ad286"}
Sep 11 18:42:06.853806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:06.853773    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" event={"ID":"41eb2eac-75fc-450f-b23e-1248fd5dffd1","Type":"ContainerStarted","Data":"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"}
Sep 11 18:42:06.854199 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:06.853843    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:06.873894 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:06.873848    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" podStartSLOduration=2.35230684 podStartE2EDuration="2.873832897s" totalImagesPullingTime="521.526057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:06.872054024 +0000 UTC m=+1596.758026513" watchObservedRunningTime="2026-09-11 18:42:06.873832897 +0000 UTC m=+1596.759805386"
Sep 11 18:42:12.864892 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:12.864864    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:42:38.537643 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.537557    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:42:38.542016 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.541994    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.545327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.545303    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-999aa4-serving-cert\""
Sep 11 18:42:38.556977 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.556945    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:42:38.592383 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.592339    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.592533 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.592399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.592533 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.592463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.592533 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.592527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcd4t\" (UniqueName: \"kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.693107 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.693062    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcd4t\" (UniqueName: \"kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.693385 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.693352    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.693474 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.693407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.693474 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.693459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.694534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.694502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.697307 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.696772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.697307 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.697262    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.701813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.701785    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcd4t\" (UniqueName: \"kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t\") pod \"maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:38.854927 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:38.854838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:39.131658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.131625    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:42:39.133237 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:42:39.133212    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5fa46e_b3aa_40a1_9a13_ba799458ff29.slice/crio-ef40239ebe633522a2d7e305b073afb29e2bd677879095b624a4af05218ef7ab WatchSource:0}: Error finding container ef40239ebe633522a2d7e305b073afb29e2bd677879095b624a4af05218ef7ab: Status 404 returned error can't find the container with id ef40239ebe633522a2d7e305b073afb29e2bd677879095b624a4af05218ef7ab
Sep 11 18:42:39.662403 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.662370    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:42:39.666607 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.666584    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.669226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.669084    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-999aa4\""
Sep 11 18:42:39.669226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.669121    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-999aa4-dockercfg-glr99\""
Sep 11 18:42:39.676433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.676411    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:42:39.703481 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.703451    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.703643 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.703619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nltqc\" (UniqueName: \"kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.804442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.804377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nltqc\" (UniqueName: \"kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.804581 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.804461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.805156 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.805133    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.812756 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.812733    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nltqc\" (UniqueName: \"kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc\") pod \"payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:39.978214 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:39.978158    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:40.022630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:40.022588    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" event={"ID":"2a5fa46e-b3aa-40a1-9a13-ba799458ff29","Type":"ContainerStarted","Data":"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"}
Sep 11 18:42:40.022806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:40.022661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" event={"ID":"2a5fa46e-b3aa-40a1-9a13-ba799458ff29","Type":"ContainerStarted","Data":"ef40239ebe633522a2d7e305b073afb29e2bd677879095b624a4af05218ef7ab"}
Sep 11 18:42:40.022806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:40.022798    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:40.043829 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:40.043785    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" podStartSLOduration=1.659030666 podStartE2EDuration="2.043768969s" totalImagesPullingTime="384.738303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:40.041242562 +0000 UTC m=+1629.927215052" watchObservedRunningTime="2026-09-11 18:42:40.043768969 +0000 UTC m=+1629.929741457"
Sep 11 18:42:40.249958 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:40.249931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:42:40.252163 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:42:40.252137    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dacb17d_fda3_4fd9_8535_79fc0e7e7673.slice/crio-e5945d3fe8288c78645e22dc67758feb1c7f20258c36ed4493427dd11ae2ad3b WatchSource:0}: Error finding container e5945d3fe8288c78645e22dc67758feb1c7f20258c36ed4493427dd11ae2ad3b: Status 404 returned error can't find the container with id e5945d3fe8288c78645e22dc67758feb1c7f20258c36ed4493427dd11ae2ad3b
Sep 11 18:42:41.029424 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:41.029389    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" event={"ID":"4dacb17d-fda3-4fd9-8535-79fc0e7e7673","Type":"ContainerStarted","Data":"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"}
Sep 11 18:42:41.029854 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:41.029449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" event={"ID":"4dacb17d-fda3-4fd9-8535-79fc0e7e7673","Type":"ContainerStarted","Data":"e5945d3fe8288c78645e22dc67758feb1c7f20258c36ed4493427dd11ae2ad3b"}
Sep 11 18:42:41.046267 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:41.046222    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" podStartSLOduration=2.046206936 podStartE2EDuration="2.046206936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:42:41.045376573 +0000 UTC m=+1630.931349063" watchObservedRunningTime="2026-09-11 18:42:41.046206936 +0000 UTC m=+1630.932179421"
Sep 11 18:42:42.035399 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:42.035369    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:42:46.034985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:46.034955    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:42:53.042309 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:42:53.042278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:43:18.174874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.174829    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:43:18.175491 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.175080    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" podUID="bdc80574-906b-4040-8b7b-8ff574470fd9" containerName="maas-api" containerID="cri-o://f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb" gracePeriod=30
Sep 11 18:43:18.248130 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.248094    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:43:18.248396 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.248368    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" podUID="28b6520a-3126-4282-abd9-839c7d653723" containerName="payload-processing" containerID="cri-o://290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe" gracePeriod=30
Sep 11 18:43:18.256947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.255839    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:43:18.256947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.256047    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" podUID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" containerName="payload-pre-processing" containerID="cri-o://454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a" gracePeriod=30
Sep 11 18:43:18.553533 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.553111    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:43:18.564784 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.564761    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:43:18.569026 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.568998    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:43:18.654709 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.654684    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hmsxc\" (UniqueName: \"kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc\") pod \"28b6520a-3126-4282-abd9-839c7d653723\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") "
Sep 11 18:43:18.654861 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.654740    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume\") pod \"28b6520a-3126-4282-abd9-839c7d653723\" (UID: \"28b6520a-3126-4282-abd9-839c7d653723\") "
Sep 11 18:43:18.655222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.655155    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28b6520a-3126-4282-abd9-839c7d653723" (UID: "28b6520a-3126-4282-abd9-839c7d653723"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:18.656766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.656746    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc" (OuterVolumeSpecName: "kube-api-access-hmsxc") pod "28b6520a-3126-4282-abd9-839c7d653723" (UID: "28b6520a-3126-4282-abd9-839c7d653723"). InnerVolumeSpecName "kube-api-access-hmsxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:18.756231 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756136    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67pc4\" (UniqueName: \"kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4\") pod \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") "
Sep 11 18:43:18.756231 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume\") pod \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\" (UID: \"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258\") "
Sep 11 18:43:18.756442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756267    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr54m\" (UniqueName: \"kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m\") pod \"bdc80574-906b-4040-8b7b-8ff574470fd9\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") "
Sep 11 18:43:18.756442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756371    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca\") pod \"bdc80574-906b-4040-8b7b-8ff574470fd9\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") "
Sep 11 18:43:18.756442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756407    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") pod \"bdc80574-906b-4040-8b7b-8ff574470fd9\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") "
Sep 11 18:43:18.756442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756440    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs\") pod \"bdc80574-906b-4040-8b7b-8ff574470fd9\" (UID: \"bdc80574-906b-4040-8b7b-8ff574470fd9\") "
Sep 11 18:43:18.756785 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756738    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" (UID: "fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:18.756785 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756756    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdc80574-906b-4040-8b7b-8ff574470fd9" (UID: "bdc80574-906b-4040-8b7b-8ff574470fd9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:18.756986 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmsxc\" (UniqueName: \"kubernetes.io/projected/28b6520a-3126-4282-abd9-839c7d653723-kube-api-access-hmsxc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.756986 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.756822    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28b6520a-3126-4282-abd9-839c7d653723-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.758465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.758429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdc80574-906b-4040-8b7b-8ff574470fd9" (UID: "bdc80574-906b-4040-8b7b-8ff574470fd9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:18.758579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.758471    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4" (OuterVolumeSpecName: "kube-api-access-67pc4") pod "fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" (UID: "fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258"). InnerVolumeSpecName "kube-api-access-67pc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:18.758579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.758498    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m" (OuterVolumeSpecName: "kube-api-access-zr54m") pod "bdc80574-906b-4040-8b7b-8ff574470fd9" (UID: "bdc80574-906b-4040-8b7b-8ff574470fd9"). InnerVolumeSpecName "kube-api-access-zr54m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:18.758579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.758532    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bdc80574-906b-4040-8b7b-8ff574470fd9" (UID: "bdc80574-906b-4040-8b7b-8ff574470fd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:18.857228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857197    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc80574-906b-4040-8b7b-8ff574470fd9-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.857228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857227    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.857407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857238    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdc80574-906b-4040-8b7b-8ff574470fd9-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.857407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857253    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67pc4\" (UniqueName: \"kubernetes.io/projected/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-kube-api-access-67pc4\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.857407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857263    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:18.857407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:18.857271    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr54m\" (UniqueName: \"kubernetes.io/projected/bdc80574-906b-4040-8b7b-8ff574470fd9-kube-api-access-zr54m\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:19.227984 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.227953    2582 generic.go:356] "Generic (PLEG): container finished" podID="28b6520a-3126-4282-abd9-839c7d653723" containerID="290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe" exitCode=0
Sep 11 18:43:19.228391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.227993    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"
Sep 11 18:43:19.228391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.228033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" event={"ID":"28b6520a-3126-4282-abd9-839c7d653723","Type":"ContainerDied","Data":"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"}
Sep 11 18:43:19.228391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.228089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr" event={"ID":"28b6520a-3126-4282-abd9-839c7d653723","Type":"ContainerDied","Data":"08f712d301a80e758288c7e69d3e1156f9c34f633d5d35ca91bc9b54fdc8783d"}
Sep 11 18:43:19.228391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.228114    2582 scope.go:117] "RemoveContainer" containerID="290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"
Sep 11 18:43:19.229511 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.229458    2582 generic.go:356] "Generic (PLEG): container finished" podID="bdc80574-906b-4040-8b7b-8ff574470fd9" containerID="f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb" exitCode=0
Sep 11 18:43:19.229595 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.229515    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"
Sep 11 18:43:19.229595 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.229545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" event={"ID":"bdc80574-906b-4040-8b7b-8ff574470fd9","Type":"ContainerDied","Data":"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"}
Sep 11 18:43:19.229703 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.229596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b" event={"ID":"bdc80574-906b-4040-8b7b-8ff574470fd9","Type":"ContainerDied","Data":"e83b4487b5e3a1ea465f50fd2270f76ad0de63d1044081c3afaf2964654b79ad"}
Sep 11 18:43:19.231014 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.230994    2582 generic.go:356] "Generic (PLEG): container finished" podID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" containerID="454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a" exitCode=0
Sep 11 18:43:19.231149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.231036    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"
Sep 11 18:43:19.231149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.231058    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" event={"ID":"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258","Type":"ContainerDied","Data":"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"}
Sep 11 18:43:19.231149 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.231092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg" event={"ID":"fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258","Type":"ContainerDied","Data":"e4c664910f13716c06fb2f029d7b31e2f20bec5b19f12bd7fc286ff0918c06ed"}
Sep 11 18:43:19.239228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.239210    2582 scope.go:117] "RemoveContainer" containerID="290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"
Sep 11 18:43:19.239533 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:19.239512    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe\": container with ID starting with 290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe not found: ID does not exist" containerID="290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"
Sep 11 18:43:19.239587 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.239546    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe"} err="failed to get container status \"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe\": rpc error: code = NotFound desc = could not find container \"290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe\": container with ID starting with 290314f16d67a03b68eaad570818f2342dea3a34a67c115923d59bb7bfa1f8fe not found: ID does not exist"
Sep 11 18:43:19.239587 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.239570    2582 scope.go:117] "RemoveContainer" containerID="f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"
Sep 11 18:43:19.247624 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.247607    2582 scope.go:117] "RemoveContainer" containerID="f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"
Sep 11 18:43:19.247882 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:19.247864    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb\": container with ID starting with f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb not found: ID does not exist" containerID="f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"
Sep 11 18:43:19.247929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.247889    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb"} err="failed to get container status \"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb\": rpc error: code = NotFound desc = could not find container \"f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb\": container with ID starting with f590eff5ad6159798d664ca02a9953c7095b3348936a8d589074b83d674fafbb not found: ID does not exist"
Sep 11 18:43:19.247929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.247904    2582 scope.go:117] "RemoveContainer" containerID="454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"
Sep 11 18:43:19.255341 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.255249    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:43:19.255387 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.255361    2582 scope.go:117] "RemoveContainer" containerID="454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"
Sep 11 18:43:19.255608 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:19.255591    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a\": container with ID starting with 454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a not found: ID does not exist" containerID="454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"
Sep 11 18:43:19.255659 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.255613    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a"} err="failed to get container status \"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a\": rpc error: code = NotFound desc = could not find container \"454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a\": container with ID starting with 454c6f5e6ce580bd002e275247040da0244281a16b376a477af397cf297f153a not found: ID does not exist"
Sep 11 18:43:19.259460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.259439    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-b63ece-78fbf6bfc5-pphmr"]
Sep 11 18:43:19.269327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.269308    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:43:19.275838 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.275818    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-b63ece-6f9b7fc4d6-w56fg"]
Sep 11 18:43:19.285452 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.285432    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:43:19.290114 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:19.290089    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-b63ece-7cd9fdf794-th65b"]
Sep 11 18:43:20.559845 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:20.559809    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b6520a-3126-4282-abd9-839c7d653723" path="/var/lib/kubelet/pods/28b6520a-3126-4282-abd9-839c7d653723/volumes"
Sep 11 18:43:20.560249 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:20.560212    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc80574-906b-4040-8b7b-8ff574470fd9" path="/var/lib/kubelet/pods/bdc80574-906b-4040-8b7b-8ff574470fd9/volumes"
Sep 11 18:43:20.560527 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:20.560514    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" path="/var/lib/kubelet/pods/fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258/volumes"
Sep 11 18:43:26.271985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:26.271953    2582 generic.go:356] "Generic (PLEG): container finished" podID="da90cf07-4257-47ea-807b-47629aaec120" containerID="2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1" exitCode=137
Sep 11 18:43:26.272450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:26.272033    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerDied","Data":"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"}
Sep 11 18:43:26.272450 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:26.272397    2582 scope.go:117] "RemoveContainer" containerID="2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"
Sep 11 18:43:27.284827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:27.284793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerStarted","Data":"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"}
Sep 11 18:43:27.285328 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:27.285039    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:43:32.492839 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.492797    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:43:32.493287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.493096    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" podUID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" containerName="maas-api" containerID="cri-o://f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998" gracePeriod=30
Sep 11 18:43:32.750031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.749965    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:43:32.775864 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.775841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") pod \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") "
Sep 11 18:43:32.775982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.775911    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrvj9\" (UniqueName: \"kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9\") pod \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") "
Sep 11 18:43:32.775982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.775980    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs\") pod \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") "
Sep 11 18:43:32.776118 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.776026    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca\") pod \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\" (UID: \"41eb2eac-75fc-450f-b23e-1248fd5dffd1\") "
Sep 11 18:43:32.776560 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.776528    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "41eb2eac-75fc-450f-b23e-1248fd5dffd1" (UID: "41eb2eac-75fc-450f-b23e-1248fd5dffd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:32.778072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.778044    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9" (OuterVolumeSpecName: "kube-api-access-vrvj9") pod "41eb2eac-75fc-450f-b23e-1248fd5dffd1" (UID: "41eb2eac-75fc-450f-b23e-1248fd5dffd1"). InnerVolumeSpecName "kube-api-access-vrvj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:32.778303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.778146    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "41eb2eac-75fc-450f-b23e-1248fd5dffd1" (UID: "41eb2eac-75fc-450f-b23e-1248fd5dffd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:32.778303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.778251    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "41eb2eac-75fc-450f-b23e-1248fd5dffd1" (UID: "41eb2eac-75fc-450f-b23e-1248fd5dffd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:32.877250 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.877216    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.877250 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.877250    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41eb2eac-75fc-450f-b23e-1248fd5dffd1-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.877555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.877265    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41eb2eac-75fc-450f-b23e-1248fd5dffd1-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:32.877555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:32.877278    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrvj9\" (UniqueName: \"kubernetes.io/projected/41eb2eac-75fc-450f-b23e-1248fd5dffd1-kube-api-access-vrvj9\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:33.314510 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.314478    2582 generic.go:356] "Generic (PLEG): container finished" podID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" containerID="f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998" exitCode=0
Sep 11 18:43:33.314690 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.314525    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"
Sep 11 18:43:33.314690 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.314525    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" event={"ID":"41eb2eac-75fc-450f-b23e-1248fd5dffd1","Type":"ContainerDied","Data":"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"}
Sep 11 18:43:33.314690 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.314652    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh" event={"ID":"41eb2eac-75fc-450f-b23e-1248fd5dffd1","Type":"ContainerDied","Data":"77da6d1aa31eec524631fa9cdc7c41b6feab1e2a2cc40d8b9df1dce0d85ad286"}
Sep 11 18:43:33.314690 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.314681    2582 scope.go:117] "RemoveContainer" containerID="f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"
Sep 11 18:43:33.324260 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.324244    2582 scope.go:117] "RemoveContainer" containerID="f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"
Sep 11 18:43:33.324547 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:33.324529    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998\": container with ID starting with f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998 not found: ID does not exist" containerID="f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"
Sep 11 18:43:33.324603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.324555    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998"} err="failed to get container status \"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998\": rpc error: code = NotFound desc = could not find container \"f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998\": container with ID starting with f5c928d928570ba101d8cc9eeff1a530d0575f1374c8116da486ecd7c3b7c998 not found: ID does not exist"
Sep 11 18:43:33.336851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.336831    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:43:33.341741 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:33.341720    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-6c4c23-5b6664fd9c-hxkvh"]
Sep 11 18:43:34.553047 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:34.553016    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" path="/var/lib/kubelet/pods/41eb2eac-75fc-450f-b23e-1248fd5dffd1/volumes"
Sep 11 18:43:38.291964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:38.291931    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:43:55.483969 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.483894    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:43:55.484384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.484124    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" podUID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" containerName="maas-api" containerID="cri-o://fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f" gracePeriod=30
Sep 11 18:43:55.554881 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.554850    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:43:55.555236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.555152    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" podUID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" containerName="payload-pre-processing" containerID="cri-o://af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047" gracePeriod=30
Sep 11 18:43:55.809355 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.809331    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:43:55.854508 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.854480    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:43:55.973021 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.972992    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume\") pod \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") "
Sep 11 18:43:55.973324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973043    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca\") pod \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") "
Sep 11 18:43:55.973324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973060    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcd4t\" (UniqueName: \"kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t\") pod \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") "
Sep 11 18:43:55.973324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs\") pod \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") "
Sep 11 18:43:55.973324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973111    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nltqc\" (UniqueName: \"kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc\") pod \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\" (UID: \"4dacb17d-fda3-4fd9-8535-79fc0e7e7673\") "
Sep 11 18:43:55.973324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973150    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls\") pod \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\" (UID: \"2a5fa46e-b3aa-40a1-9a13-ba799458ff29\") "
Sep 11 18:43:55.973637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973445    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2a5fa46e-b3aa-40a1-9a13-ba799458ff29" (UID: "2a5fa46e-b3aa-40a1-9a13-ba799458ff29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:55.973637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.973467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dacb17d-fda3-4fd9-8535-79fc0e7e7673" (UID: "4dacb17d-fda3-4fd9-8535-79fc0e7e7673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:43:55.975318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.975292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t" (OuterVolumeSpecName: "kube-api-access-vcd4t") pod "2a5fa46e-b3aa-40a1-9a13-ba799458ff29" (UID: "2a5fa46e-b3aa-40a1-9a13-ba799458ff29"). InnerVolumeSpecName "kube-api-access-vcd4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:55.975421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.975369    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2a5fa46e-b3aa-40a1-9a13-ba799458ff29" (UID: "2a5fa46e-b3aa-40a1-9a13-ba799458ff29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:55.975421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.975386    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc" (OuterVolumeSpecName: "kube-api-access-nltqc") pod "4dacb17d-fda3-4fd9-8535-79fc0e7e7673" (UID: "4dacb17d-fda3-4fd9-8535-79fc0e7e7673"). InnerVolumeSpecName "kube-api-access-nltqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:43:55.975421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:55.975389    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2a5fa46e-b3aa-40a1-9a13-ba799458ff29" (UID: "2a5fa46e-b3aa-40a1-9a13-ba799458ff29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074081    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074106    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074117    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074126    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcd4t\" (UniqueName: \"kubernetes.io/projected/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-kube-api-access-vcd4t\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074134    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a5fa46e-b3aa-40a1-9a13-ba799458ff29-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.074150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.074142    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nltqc\" (UniqueName: \"kubernetes.io/projected/4dacb17d-fda3-4fd9-8535-79fc0e7e7673-kube-api-access-nltqc\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:43:56.435613 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.435577    2582 generic.go:356] "Generic (PLEG): container finished" podID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" containerID="af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047" exitCode=0
Sep 11 18:43:56.435780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.435635    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"
Sep 11 18:43:56.435780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.435663    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" event={"ID":"4dacb17d-fda3-4fd9-8535-79fc0e7e7673","Type":"ContainerDied","Data":"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"}
Sep 11 18:43:56.435780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.435746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd" event={"ID":"4dacb17d-fda3-4fd9-8535-79fc0e7e7673","Type":"ContainerDied","Data":"e5945d3fe8288c78645e22dc67758feb1c7f20258c36ed4493427dd11ae2ad3b"}
Sep 11 18:43:56.435780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.435773    2582 scope.go:117] "RemoveContainer" containerID="af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"
Sep 11 18:43:56.437122 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.437100    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" containerID="fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f" exitCode=0
Sep 11 18:43:56.437222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.437127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" event={"ID":"2a5fa46e-b3aa-40a1-9a13-ba799458ff29","Type":"ContainerDied","Data":"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"}
Sep 11 18:43:56.437222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.437166    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"
Sep 11 18:43:56.437222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.437204    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc" event={"ID":"2a5fa46e-b3aa-40a1-9a13-ba799458ff29","Type":"ContainerDied","Data":"ef40239ebe633522a2d7e305b073afb29e2bd677879095b624a4af05218ef7ab"}
Sep 11 18:43:56.446367 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.446350    2582 scope.go:117] "RemoveContainer" containerID="af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"
Sep 11 18:43:56.446665 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:56.446638    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047\": container with ID starting with af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047 not found: ID does not exist" containerID="af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"
Sep 11 18:43:56.446781 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.446675    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047"} err="failed to get container status \"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047\": rpc error: code = NotFound desc = could not find container \"af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047\": container with ID starting with af00346dcf82ea9b789d49db366ac058460275114aefcc68ad7aa0992ba6d047 not found: ID does not exist"
Sep 11 18:43:56.446781 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.446692    2582 scope.go:117] "RemoveContainer" containerID="fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"
Sep 11 18:43:56.455520 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.455497    2582 scope.go:117] "RemoveContainer" containerID="fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"
Sep 11 18:43:56.455840 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:43:56.455818    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f\": container with ID starting with fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f not found: ID does not exist" containerID="fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"
Sep 11 18:43:56.455901 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.455849    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f"} err="failed to get container status \"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f\": rpc error: code = NotFound desc = could not find container \"fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f\": container with ID starting with fdd7cad4c16a367be2371494fb396753b64595ae3ba12417e3d55409ca918a1f not found: ID does not exist"
Sep 11 18:43:56.457076 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.457057    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:43:56.463395 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.463374    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-999aa4-58b6bb66cd-f8ldd"]
Sep 11 18:43:56.473980 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.473961    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:43:56.478144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.478125    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-999aa4-79dcf6b4f6-zblxc"]
Sep 11 18:43:56.554884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.554854    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" path="/var/lib/kubelet/pods/2a5fa46e-b3aa-40a1-9a13-ba799458ff29/volumes"
Sep 11 18:43:56.555266 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:43:56.555251    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" path="/var/lib/kubelet/pods/4dacb17d-fda3-4fd9-8535-79fc0e7e7673/volumes"
Sep 11 18:44:09.263690 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.263655    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"]
Sep 11 18:44:09.264051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264018    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" containerName="payload-pre-processing"
Sep 11 18:44:09.264051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264030    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" containerName="payload-pre-processing"
Sep 11 18:44:09.264051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264039    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" containerName="payload-pre-processing"
Sep 11 18:44:09.264051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264044    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" containerName="payload-pre-processing"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264054    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc80574-906b-4040-8b7b-8ff574470fd9" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264060    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc80574-906b-4040-8b7b-8ff574470fd9" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264068    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264073    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264082    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b6520a-3126-4282-abd9-839c7d653723" containerName="payload-processing"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264090    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b6520a-3126-4282-abd9-839c7d653723" containerName="payload-processing"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264117    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264123    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264196    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5fa46e-b3aa-40a1-9a13-ba799458ff29" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264205    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="41eb2eac-75fc-450f-b23e-1248fd5dffd1" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264211    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc80574-906b-4040-8b7b-8ff574470fd9" containerName="maas-api"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264217    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dacb17d-fda3-4fd9-8535-79fc0e7e7673" containerName="payload-pre-processing"
Sep 11 18:44:09.264220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264223    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4fe3f3-bdc9-4e59-88c7-65b6a2e57258" containerName="payload-pre-processing"
Sep 11 18:44:09.264688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.264228    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b6520a-3126-4282-abd9-839c7d653723" containerName="payload-processing"
Sep 11 18:44:09.267256 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.267234    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:09.269790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.269764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:44:09.273255 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.273235    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"]
Sep 11 18:44:09.398989 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.398943    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5\" (UID: \"f7441369-7c7b-4138-ba15-ca991f6bdf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:09.499592 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.499564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5\" (UID: \"f7441369-7c7b-4138-ba15-ca991f6bdf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:09.500180 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.500146    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5\" (UID: \"f7441369-7c7b-4138-ba15-ca991f6bdf7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:09.579018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.578943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:09.847561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:09.847536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"]
Sep 11 18:44:09.849881 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:44:09.849852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7441369_7c7b_4138_ba15_ca991f6bdf7b.slice/crio-2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65 WatchSource:0}: Error finding container 2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65: Status 404 returned error can't find the container with id 2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65
Sep 11 18:44:10.511644 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:10.511609    2582 generic.go:356] "Generic (PLEG): container finished" podID="f7441369-7c7b-4138-ba15-ca991f6bdf7b" containerID="7cfe8a0e3328268bf8a40eac712987ad1ee443fd3ff2580af09eb14eba141b93" exitCode=0
Sep 11 18:44:10.511940 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:10.511661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5" event={"ID":"f7441369-7c7b-4138-ba15-ca991f6bdf7b","Type":"ContainerDied","Data":"7cfe8a0e3328268bf8a40eac712987ad1ee443fd3ff2580af09eb14eba141b93"}
Sep 11 18:44:10.511940 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:10.511710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5" event={"ID":"f7441369-7c7b-4138-ba15-ca991f6bdf7b","Type":"ContainerStarted","Data":"2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65"}
Sep 11 18:44:11.650146 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:11.650120    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:11.718823 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:11.718794    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca\") pod \"f7441369-7c7b-4138-ba15-ca991f6bdf7b\" (UID: \"f7441369-7c7b-4138-ba15-ca991f6bdf7b\") "
Sep 11 18:44:11.719147 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:11.719126    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7441369-7c7b-4138-ba15-ca991f6bdf7b" (UID: "f7441369-7c7b-4138-ba15-ca991f6bdf7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:11.819889 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:11.819812    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7441369-7c7b-4138-ba15-ca991f6bdf7b-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:12.522710 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:12.522680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"
Sep 11 18:44:12.522886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:12.522712    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5" event={"ID":"f7441369-7c7b-4138-ba15-ca991f6bdf7b","Type":"ContainerDied","Data":"2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65"}
Sep 11 18:44:12.522886 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:12.522761    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3390273b914508d74290850292f279e2529f57735a5b2eeb2fc4fe3c631d65"
Sep 11 18:44:14.611884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.611848    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:44:14.612382 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.612221    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" podUID="451ca7f9-9cbd-4377-bff4-0801c1a75249" containerName="maas-api" containerID="cri-o://632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6" gracePeriod=30
Sep 11 18:44:14.670687 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.670655    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:44:14.670914 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.670888    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" podUID="ad0abde1-71da-47c5-a6ad-ab5431864d86" containerName="payload-processing" containerID="cri-o://60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a" gracePeriod=30
Sep 11 18:44:14.688460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.678366    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:44:14.688460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.678566    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" podUID="c879beeb-e297-4e19-96c7-e5aa69255154" containerName="payload-pre-processing" containerID="cri-o://db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7" gracePeriod=30
Sep 11 18:44:14.987287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:14.987257    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:44:15.000408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.000382    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:44:15.002418 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.002403    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:44:15.045138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045113    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") pod \"451ca7f9-9cbd-4377-bff4-0801c1a75249\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") "
Sep 11 18:44:15.045282 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045154    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp9zw\" (UniqueName: \"kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw\") pod \"c879beeb-e297-4e19-96c7-e5aa69255154\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") "
Sep 11 18:44:15.045282 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045261    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume\") pod \"ad0abde1-71da-47c5-a6ad-ab5431864d86\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") "
Sep 11 18:44:15.045369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045310    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tklmg\" (UniqueName: \"kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg\") pod \"451ca7f9-9cbd-4377-bff4-0801c1a75249\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") "
Sep 11 18:44:15.045369 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045352    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca\") pod \"451ca7f9-9cbd-4377-bff4-0801c1a75249\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") "
Sep 11 18:44:15.045489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045395    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs\") pod \"451ca7f9-9cbd-4377-bff4-0801c1a75249\" (UID: \"451ca7f9-9cbd-4377-bff4-0801c1a75249\") "
Sep 11 18:44:15.045489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045431    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume\") pod \"c879beeb-e297-4e19-96c7-e5aa69255154\" (UID: \"c879beeb-e297-4e19-96c7-e5aa69255154\") "
Sep 11 18:44:15.045489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045455    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ht5j9\" (UniqueName: \"kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9\") pod \"ad0abde1-71da-47c5-a6ad-ab5431864d86\" (UID: \"ad0abde1-71da-47c5-a6ad-ab5431864d86\") "
Sep 11 18:44:15.046069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045652    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad0abde1-71da-47c5-a6ad-ab5431864d86" (UID: "ad0abde1-71da-47c5-a6ad-ab5431864d86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:15.046069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045694    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "451ca7f9-9cbd-4377-bff4-0801c1a75249" (UID: "451ca7f9-9cbd-4377-bff4-0801c1a75249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:15.046069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.045761    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad0abde1-71da-47c5-a6ad-ab5431864d86-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.046069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.046035    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c879beeb-e297-4e19-96c7-e5aa69255154" (UID: "c879beeb-e297-4e19-96c7-e5aa69255154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:15.047757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.047727    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg" (OuterVolumeSpecName: "kube-api-access-tklmg") pod "451ca7f9-9cbd-4377-bff4-0801c1a75249" (UID: "451ca7f9-9cbd-4377-bff4-0801c1a75249"). InnerVolumeSpecName "kube-api-access-tklmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:15.047882 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.047862    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9" (OuterVolumeSpecName: "kube-api-access-ht5j9") pod "ad0abde1-71da-47c5-a6ad-ab5431864d86" (UID: "ad0abde1-71da-47c5-a6ad-ab5431864d86"). InnerVolumeSpecName "kube-api-access-ht5j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:15.047944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.047905    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "451ca7f9-9cbd-4377-bff4-0801c1a75249" (UID: "451ca7f9-9cbd-4377-bff4-0801c1a75249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:15.048003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.047953    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw" (OuterVolumeSpecName: "kube-api-access-jp9zw") pod "c879beeb-e297-4e19-96c7-e5aa69255154" (UID: "c879beeb-e297-4e19-96c7-e5aa69255154"). InnerVolumeSpecName "kube-api-access-jp9zw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:15.048435 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.048410    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "451ca7f9-9cbd-4377-bff4-0801c1a75249" (UID: "451ca7f9-9cbd-4377-bff4-0801c1a75249"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147055    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/451ca7f9-9cbd-4377-bff4-0801c1a75249-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147084    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147094    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c879beeb-e297-4e19-96c7-e5aa69255154-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147104    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht5j9\" (UniqueName: \"kubernetes.io/projected/ad0abde1-71da-47c5-a6ad-ab5431864d86-kube-api-access-ht5j9\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147112    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/451ca7f9-9cbd-4377-bff4-0801c1a75249-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147120    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9zw\" (UniqueName: \"kubernetes.io/projected/c879beeb-e297-4e19-96c7-e5aa69255154-kube-api-access-jp9zw\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.147138 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.147128    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tklmg\" (UniqueName: \"kubernetes.io/projected/451ca7f9-9cbd-4377-bff4-0801c1a75249-kube-api-access-tklmg\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:15.539301 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.539269    2582 generic.go:356] "Generic (PLEG): container finished" podID="ad0abde1-71da-47c5-a6ad-ab5431864d86" containerID="60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a" exitCode=0
Sep 11 18:44:15.539488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.539312    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"
Sep 11 18:44:15.539488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.539354    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" event={"ID":"ad0abde1-71da-47c5-a6ad-ab5431864d86","Type":"ContainerDied","Data":"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"}
Sep 11 18:44:15.539488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.539408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9" event={"ID":"ad0abde1-71da-47c5-a6ad-ab5431864d86","Type":"ContainerDied","Data":"e7682e876bf63fdda69f32de2d4e95fd239bd6d03ecf1fcd7543e2e05096fb52"}
Sep 11 18:44:15.539488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.539441    2582 scope.go:117] "RemoveContainer" containerID="60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"
Sep 11 18:44:15.540888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.540866    2582 generic.go:356] "Generic (PLEG): container finished" podID="451ca7f9-9cbd-4377-bff4-0801c1a75249" containerID="632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6" exitCode=0
Sep 11 18:44:15.540990 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.540896    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" event={"ID":"451ca7f9-9cbd-4377-bff4-0801c1a75249","Type":"ContainerDied","Data":"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"}
Sep 11 18:44:15.540990 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.540949    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk" event={"ID":"451ca7f9-9cbd-4377-bff4-0801c1a75249","Type":"ContainerDied","Data":"5731fd32f67350a6964edd41f77b0b136ea56f23fa66ad47e3c9cc52cfa411e0"}
Sep 11 18:44:15.540990 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.540906    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"
Sep 11 18:44:15.542372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.542351    2582 generic.go:356] "Generic (PLEG): container finished" podID="c879beeb-e297-4e19-96c7-e5aa69255154" containerID="db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7" exitCode=0
Sep 11 18:44:15.542513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.542377    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" event={"ID":"c879beeb-e297-4e19-96c7-e5aa69255154","Type":"ContainerDied","Data":"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"}
Sep 11 18:44:15.542513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.542386    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"
Sep 11 18:44:15.542513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.542408    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4" event={"ID":"c879beeb-e297-4e19-96c7-e5aa69255154","Type":"ContainerDied","Data":"e7daad12cf0374c96b3949ab01d0a0c87cbb8fafb8ce26160218744569c95ada"}
Sep 11 18:44:15.549933 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.549890    2582 scope.go:117] "RemoveContainer" containerID="60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"
Sep 11 18:44:15.550314 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:15.550159    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a\": container with ID starting with 60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a not found: ID does not exist" containerID="60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"
Sep 11 18:44:15.550417 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.550312    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a"} err="failed to get container status \"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a\": rpc error: code = NotFound desc = could not find container \"60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a\": container with ID starting with 60ddc0759b2a3fba4f0cf3efb2946714db72fcf97a282cfb4f0afb9d993f940a not found: ID does not exist"
Sep 11 18:44:15.550417 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.550334    2582 scope.go:117] "RemoveContainer" containerID="632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"
Sep 11 18:44:15.558424 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.558408    2582 scope.go:117] "RemoveContainer" containerID="632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"
Sep 11 18:44:15.558679 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:15.558661    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6\": container with ID starting with 632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6 not found: ID does not exist" containerID="632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"
Sep 11 18:44:15.558766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.558684    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6"} err="failed to get container status \"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6\": rpc error: code = NotFound desc = could not find container \"632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6\": container with ID starting with 632908f484bd3f2401aee283f829d0d02c42bf40900f7930e5a9a63078c192a6 not found: ID does not exist"
Sep 11 18:44:15.558766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.558699    2582 scope.go:117] "RemoveContainer" containerID="db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"
Sep 11 18:44:15.564217 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.564197    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:44:15.567028 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.567011    2582 scope.go:117] "RemoveContainer" containerID="db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"
Sep 11 18:44:15.567345 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:15.567324    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7\": container with ID starting with db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7 not found: ID does not exist" containerID="db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"
Sep 11 18:44:15.567408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.567351    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7"} err="failed to get container status \"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7\": rpc error: code = NotFound desc = could not find container \"db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7\": container with ID starting with db58882477cbfa259afd6bc58547c87933c51d5b31c2cc4c048e256498dc65f7 not found: ID does not exist"
Sep 11 18:44:15.567695 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.567679    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-93d67f-8d49f645f-zkgr9"]
Sep 11 18:44:15.585632 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.585610    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:44:15.594776 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.594756    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-93d67f-854d78b654-6zpx4"]
Sep 11 18:44:15.605866 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.605847    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:44:15.610820 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:15.610798    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-93d67f-84dbbbcc8-5pgpk"]
Sep 11 18:44:16.552917 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:16.552876    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="451ca7f9-9cbd-4377-bff4-0801c1a75249" path="/var/lib/kubelet/pods/451ca7f9-9cbd-4377-bff4-0801c1a75249/volumes"
Sep 11 18:44:16.553428 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:16.553400    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad0abde1-71da-47c5-a6ad-ab5431864d86" path="/var/lib/kubelet/pods/ad0abde1-71da-47c5-a6ad-ab5431864d86/volumes"
Sep 11 18:44:16.553775 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:16.553754    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c879beeb-e297-4e19-96c7-e5aa69255154" path="/var/lib/kubelet/pods/c879beeb-e297-4e19-96c7-e5aa69255154/volumes"
Sep 11 18:44:22.443136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443103    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"]
Sep 11 18:44:22.443529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443505    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="451ca7f9-9cbd-4377-bff4-0801c1a75249" containerName="maas-api"
Sep 11 18:44:22.443529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443518    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="451ca7f9-9cbd-4377-bff4-0801c1a75249" containerName="maas-api"
Sep 11 18:44:22.443529 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443531    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c879beeb-e297-4e19-96c7-e5aa69255154" containerName="payload-pre-processing"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443536    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c879beeb-e297-4e19-96c7-e5aa69255154" containerName="payload-pre-processing"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443552    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7441369-7c7b-4138-ba15-ca991f6bdf7b" containerName="revoke-keys"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443558    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7441369-7c7b-4138-ba15-ca991f6bdf7b" containerName="revoke-keys"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443564    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad0abde1-71da-47c5-a6ad-ab5431864d86" containerName="payload-processing"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443570    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad0abde1-71da-47c5-a6ad-ab5431864d86" containerName="payload-processing"
Sep 11 18:44:22.443636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443632    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c879beeb-e297-4e19-96c7-e5aa69255154" containerName="payload-pre-processing"
Sep 11 18:44:22.443852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443640    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad0abde1-71da-47c5-a6ad-ab5431864d86" containerName="payload-processing"
Sep 11 18:44:22.443852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443647    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="451ca7f9-9cbd-4377-bff4-0801c1a75249" containerName="maas-api"
Sep 11 18:44:22.443852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.443654    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7441369-7c7b-4138-ba15-ca991f6bdf7b" containerName="revoke-keys"
Sep 11 18:44:22.448228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.448210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:22.450674 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.450655    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:44:22.459594 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.459571    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"]
Sep 11 18:44:22.515286 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.515263    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc\" (UID: \"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:22.616392 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.616362    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc\" (UID: \"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:22.616953 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.616935    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc\" (UID: \"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:22.757814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:22.757750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:23.040962 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.040931    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"]
Sep 11 18:44:23.042402 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:44:23.042374    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e107ed4_3fb8_40ae_949d_1db3a0af1d6f.slice/crio-724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1 WatchSource:0}: Error finding container 724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1: Status 404 returned error can't find the container with id 724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1
Sep 11 18:44:23.481532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.481499    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:44:23.481912 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.481701    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" podUID="203b723e-af7f-4949-a05e-53a5a99d7fc3" containerName="payload-processing" containerID="cri-o://dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938" gracePeriod=30
Sep 11 18:44:23.502294 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.502268    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:44:23.502515 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.502484    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" podUID="428aa947-530b-4eb7-84db-b39697612f30" containerName="payload-pre-processing" containerID="cri-o://df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7" gracePeriod=30
Sep 11 18:44:23.583300 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.583268    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc" event={"ID":"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f","Type":"ContainerStarted","Data":"fd9326974131961df2d528ba6e9bd15c53ac5c209aa021c9a9d7dee635159abc"}
Sep 11 18:44:23.583423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.583327    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc" event={"ID":"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f","Type":"ContainerStarted","Data":"724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1"}
Sep 11 18:44:23.603053 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.603006    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc" podStartSLOduration=1.602988276 podStartE2EDuration="1.602988276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:44:23.601643142 +0000 UTC m=+1733.487615632" watchObservedRunningTime="2026-09-11 18:44:23.602988276 +0000 UTC m=+1733.488960764"
Sep 11 18:44:23.740088 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.740052    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:44:23.796555 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.796476    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:44:23.825235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.825204    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"]
Sep 11 18:44:23.826532 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.826507    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tg6j\" (UniqueName: \"kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j\") pod \"428aa947-530b-4eb7-84db-b39697612f30\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") "
Sep 11 18:44:23.826748 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.826724    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume\") pod \"203b723e-af7f-4949-a05e-53a5a99d7fc3\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") "
Sep 11 18:44:23.826848 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.826829    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume\") pod \"428aa947-530b-4eb7-84db-b39697612f30\" (UID: \"428aa947-530b-4eb7-84db-b39697612f30\") "
Sep 11 18:44:23.826911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.826873    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-td4v2\" (UniqueName: \"kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2\") pod \"203b723e-af7f-4949-a05e-53a5a99d7fc3\" (UID: \"203b723e-af7f-4949-a05e-53a5a99d7fc3\") "
Sep 11 18:44:23.827094 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.827068    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "203b723e-af7f-4949-a05e-53a5a99d7fc3" (UID: "203b723e-af7f-4949-a05e-53a5a99d7fc3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:23.827163 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.827120    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c8b274-d7rl5"]
Sep 11 18:44:23.827245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.827196    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/203b723e-af7f-4949-a05e-53a5a99d7fc3-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:23.827322 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.827298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "428aa947-530b-4eb7-84db-b39697612f30" (UID: "428aa947-530b-4eb7-84db-b39697612f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:23.828610 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.828579    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j" (OuterVolumeSpecName: "kube-api-access-8tg6j") pod "428aa947-530b-4eb7-84db-b39697612f30" (UID: "428aa947-530b-4eb7-84db-b39697612f30"). InnerVolumeSpecName "kube-api-access-8tg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:23.828934 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.828915    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2" (OuterVolumeSpecName: "kube-api-access-td4v2") pod "203b723e-af7f-4949-a05e-53a5a99d7fc3" (UID: "203b723e-af7f-4949-a05e-53a5a99d7fc3"). InnerVolumeSpecName "kube-api-access-td4v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:23.927740 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.927715    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/428aa947-530b-4eb7-84db-b39697612f30-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:23.927740 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.927738    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td4v2\" (UniqueName: \"kubernetes.io/projected/203b723e-af7f-4949-a05e-53a5a99d7fc3-kube-api-access-td4v2\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:23.927881 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:23.927749    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tg6j\" (UniqueName: \"kubernetes.io/projected/428aa947-530b-4eb7-84db-b39697612f30-kube-api-access-8tg6j\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:24.554293 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.554258    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7441369-7c7b-4138-ba15-ca991f6bdf7b" path="/var/lib/kubelet/pods/f7441369-7c7b-4138-ba15-ca991f6bdf7b/volumes"
Sep 11 18:44:24.590448 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.590413    2582 generic.go:356] "Generic (PLEG): container finished" podID="203b723e-af7f-4949-a05e-53a5a99d7fc3" containerID="dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938" exitCode=0
Sep 11 18:44:24.590561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.590467    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"
Sep 11 18:44:24.590561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.590490    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" event={"ID":"203b723e-af7f-4949-a05e-53a5a99d7fc3","Type":"ContainerDied","Data":"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"}
Sep 11 18:44:24.590561 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.590545    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz" event={"ID":"203b723e-af7f-4949-a05e-53a5a99d7fc3","Type":"ContainerDied","Data":"a53e1eadcbea67c7661eaaf8f0e2d5a8c156ce7fab5ddc20efd6aba375ac4aa8"}
Sep 11 18:44:24.590696 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.590566    2582 scope.go:117] "RemoveContainer" containerID="dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"
Sep 11 18:44:24.595259 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.594942    2582 generic.go:356] "Generic (PLEG): container finished" podID="5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" containerID="fd9326974131961df2d528ba6e9bd15c53ac5c209aa021c9a9d7dee635159abc" exitCode=0
Sep 11 18:44:24.595259 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.595041    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc" event={"ID":"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f","Type":"ContainerDied","Data":"fd9326974131961df2d528ba6e9bd15c53ac5c209aa021c9a9d7dee635159abc"}
Sep 11 18:44:24.600398 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.600367    2582 generic.go:356] "Generic (PLEG): container finished" podID="428aa947-530b-4eb7-84db-b39697612f30" containerID="df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7" exitCode=0
Sep 11 18:44:24.600483 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.600418    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"
Sep 11 18:44:24.600612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.600421    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" event={"ID":"428aa947-530b-4eb7-84db-b39697612f30","Type":"ContainerDied","Data":"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"}
Sep 11 18:44:24.600981 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.600963    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd" event={"ID":"428aa947-530b-4eb7-84db-b39697612f30","Type":"ContainerDied","Data":"409e8dba7c69e7c25db57edffe7d063d01d88e65b1aa492af80501791a8cc42d"}
Sep 11 18:44:24.605897 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.605882    2582 scope.go:117] "RemoveContainer" containerID="dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"
Sep 11 18:44:24.606182 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:24.606151    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938\": container with ID starting with dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938 not found: ID does not exist" containerID="dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"
Sep 11 18:44:24.606227 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.606193    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938"} err="failed to get container status \"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938\": rpc error: code = NotFound desc = could not find container \"dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938\": container with ID starting with dc456d91e7e865394b434b9468ba5f255c2317a02d335b436cef14a36db4d938 not found: ID does not exist"
Sep 11 18:44:24.606227 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.606215    2582 scope.go:117] "RemoveContainer" containerID="df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"
Sep 11 18:44:24.615301 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.615262    2582 scope.go:117] "RemoveContainer" containerID="df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"
Sep 11 18:44:24.615543 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:24.615525    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7\": container with ID starting with df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7 not found: ID does not exist" containerID="df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"
Sep 11 18:44:24.615603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.615550    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7"} err="failed to get container status \"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7\": rpc error: code = NotFound desc = could not find container \"df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7\": container with ID starting with df19fcdbf39664dd5bb74caac84bbab2845d64f58f42b8774c27c8710b1144e7 not found: ID does not exist"
Sep 11 18:44:24.616493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.616472    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:44:24.632068 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.632040    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-c8b274-655b68d997-kslxz"]
Sep 11 18:44:24.685351 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.685327    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:44:24.692315 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:24.692291    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-c8b274-5dcc6f8678-lt7rd"]
Sep 11 18:44:25.734248 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:25.734226    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:25.844407 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:25.844377    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca\") pod \"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f\" (UID: \"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f\") "
Sep 11 18:44:25.844774 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:25.844749    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" (UID: "5e107ed4-3fb8-40ae-949d-1db3a0af1d6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:25.945500 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:25.945474    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:26.552890 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:26.552859    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="203b723e-af7f-4949-a05e-53a5a99d7fc3" path="/var/lib/kubelet/pods/203b723e-af7f-4949-a05e-53a5a99d7fc3/volumes"
Sep 11 18:44:26.553224 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:26.553210    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428aa947-530b-4eb7-84db-b39697612f30" path="/var/lib/kubelet/pods/428aa947-530b-4eb7-84db-b39697612f30/volumes"
Sep 11 18:44:26.613673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:26.613637    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc" event={"ID":"5e107ed4-3fb8-40ae-949d-1db3a0af1d6f","Type":"ContainerDied","Data":"724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1"}
Sep 11 18:44:26.613812 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:26.613693    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724305cc0cedffd9bc88fa91e7d8b20c18accff0f1097bccfbf2c217bf8c32f1"
Sep 11 18:44:26.613812 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:26.613651    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"
Sep 11 18:44:30.965185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:30.965143    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:44:30.965630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:30.965347    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy" containerID="cri-o://8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445" gracePeriod=30
Sep 11 18:44:33.257283 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.257250    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:44:33.257730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.257506    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" podUID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" containerName="payload-processing" containerID="cri-o://6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57" gracePeriod=30
Sep 11 18:44:33.268523 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.268495    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:44:33.268736 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.268708    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" podUID="3b0d8351-e77f-4566-854b-30897f10f78f" containerName="payload-pre-processing" containerID="cri-o://0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb" gracePeriod=30
Sep 11 18:44:33.444151 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.444092    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"]
Sep 11 18:44:33.446003 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.445979    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-e6454f-mq5rc"]
Sep 11 18:44:33.540242 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.540221    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:44:33.542724 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.542705    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:44:33.613063 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613033    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lw9dk\" (UniqueName: \"kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk\") pod \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") "
Sep 11 18:44:33.613281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613138    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq4kq\" (UniqueName: \"kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq\") pod \"3b0d8351-e77f-4566-854b-30897f10f78f\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") "
Sep 11 18:44:33.613281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613207    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume\") pod \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\" (UID: \"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e\") "
Sep 11 18:44:33.613281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613264    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume\") pod \"3b0d8351-e77f-4566-854b-30897f10f78f\" (UID: \"3b0d8351-e77f-4566-854b-30897f10f78f\") "
Sep 11 18:44:33.613670 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b0d8351-e77f-4566-854b-30897f10f78f" (UID: "3b0d8351-e77f-4566-854b-30897f10f78f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:33.613670 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.613651    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" (UID: "00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:33.615318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.615297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq" (OuterVolumeSpecName: "kube-api-access-qq4kq") pod "3b0d8351-e77f-4566-854b-30897f10f78f" (UID: "3b0d8351-e77f-4566-854b-30897f10f78f"). InnerVolumeSpecName "kube-api-access-qq4kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:33.615412 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.615324    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk" (OuterVolumeSpecName: "kube-api-access-lw9dk") pod "00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" (UID: "00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e"). InnerVolumeSpecName "kube-api-access-lw9dk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:33.647550 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.647525    2582 generic.go:356] "Generic (PLEG): container finished" podID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" containerID="6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57" exitCode=0
Sep 11 18:44:33.647649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.647569    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"
Sep 11 18:44:33.647649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.647618    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" event={"ID":"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e","Type":"ContainerDied","Data":"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"}
Sep 11 18:44:33.647765 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.647679    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p" event={"ID":"00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e","Type":"ContainerDied","Data":"a8c66720c39cf01fc2fc71fb3928be32684d4c047863a127b3372b4e2be12105"}
Sep 11 18:44:33.647765 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.647712    2582 scope.go:117] "RemoveContainer" containerID="6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"
Sep 11 18:44:33.648933 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.648911    2582 generic.go:356] "Generic (PLEG): container finished" podID="3b0d8351-e77f-4566-854b-30897f10f78f" containerID="0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb" exitCode=0
Sep 11 18:44:33.649021 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.648959    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" event={"ID":"3b0d8351-e77f-4566-854b-30897f10f78f","Type":"ContainerDied","Data":"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"}
Sep 11 18:44:33.649021 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.648963    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"
Sep 11 18:44:33.649021 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.648995    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq" event={"ID":"3b0d8351-e77f-4566-854b-30897f10f78f","Type":"ContainerDied","Data":"b11f95d03950b4c0a987b7c4575a2690da6d0860ff164f2ea21d516bddde582a"}
Sep 11 18:44:33.657078 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.657064    2582 scope.go:117] "RemoveContainer" containerID="6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"
Sep 11 18:44:33.657369 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:33.657349    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57\": container with ID starting with 6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57 not found: ID does not exist" containerID="6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"
Sep 11 18:44:33.657442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.657375    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57"} err="failed to get container status \"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57\": rpc error: code = NotFound desc = could not find container \"6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57\": container with ID starting with 6d793b9868873b8cd9346bd30b5a5867321da2bdcee755ddb29c6fbea9597a57 not found: ID does not exist"
Sep 11 18:44:33.657442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.657392    2582 scope.go:117] "RemoveContainer" containerID="0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"
Sep 11 18:44:33.664899 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.664880    2582 scope.go:117] "RemoveContainer" containerID="0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"
Sep 11 18:44:33.665126 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:33.665108    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb\": container with ID starting with 0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb not found: ID does not exist" containerID="0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"
Sep 11 18:44:33.665207 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.665136    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb"} err="failed to get container status \"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb\": rpc error: code = NotFound desc = could not find container \"0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb\": container with ID starting with 0412249ad0c30843431770144a223d02b5036a5d616e0fca1c27c8d246e36feb not found: ID does not exist"
Sep 11 18:44:33.669832 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.669810    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:44:33.673970 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.673949    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-e6454f-c586669f-2tr8p"]
Sep 11 18:44:33.684415 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.684395    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:44:33.689056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.689026    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-e6454f-7db8597f8-dgpgq"]
Sep 11 18:44:33.714052 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.714035    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw9dk\" (UniqueName: \"kubernetes.io/projected/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-kube-api-access-lw9dk\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.714052 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.714054    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq4kq\" (UniqueName: \"kubernetes.io/projected/3b0d8351-e77f-4566-854b-30897f10f78f-kube-api-access-qq4kq\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.714197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.714063    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:33.714197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:33.714072    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b0d8351-e77f-4566-854b-30897f10f78f-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:34.553232 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:34.553149    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" path="/var/lib/kubelet/pods/00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e/volumes"
Sep 11 18:44:34.553569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:34.553523    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b0d8351-e77f-4566-854b-30897f10f78f" path="/var/lib/kubelet/pods/3b0d8351-e77f-4566-854b-30897f10f78f/volumes"
Sep 11 18:44:34.553800 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:34.553787    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" path="/var/lib/kubelet/pods/5e107ed4-3fb8-40ae-949d-1db3a0af1d6f/volumes"
Sep 11 18:44:36.210910 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.210885    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:44:36.335414 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335333    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zknvr\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335414 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335380    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335414 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335401    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335463    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335529    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335600    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335625    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs\") pod \"86aa2cca-2673-488d-83f3-f198be21d6bb\" (UID: \"86aa2cca-2673-488d-83f3-f198be21d6bb\") "
Sep 11 18:44:36.335689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335662    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:36.336072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335836    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:36.336072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335914    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:36.336072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335979    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-data\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.336072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.335998    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.336072 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.336008    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:44:36.336332 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.336110    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:36.337587 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.337561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr" (OuterVolumeSpecName: "kube-api-access-zknvr") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "kube-api-access-zknvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:36.337871 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.337852    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:44:36.337942 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.337918    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:44:36.338024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.338005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "86aa2cca-2673-488d-83f3-f198be21d6bb" (UID: "86aa2cca-2673-488d-83f3-f198be21d6bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:44:36.437190 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437138    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86aa2cca-2673-488d-83f3-f198be21d6bb-istiod-ca-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437190 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437191    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-envoy\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437204    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-credential-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437218    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-podinfo\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437231    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86aa2cca-2673-488d-83f3-f198be21d6bb-workload-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437245    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zknvr\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-kube-api-access-zknvr\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.437377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.437253    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86aa2cca-2673-488d-83f3-f198be21d6bb-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:44:36.665124 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.665089    2582 generic.go:356] "Generic (PLEG): container finished" podID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerID="8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445" exitCode=0
Sep 11 18:44:36.665310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.665139    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"
Sep 11 18:44:36.665310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.665152    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerDied","Data":"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"}
Sep 11 18:44:36.665310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.665222    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f" event={"ID":"86aa2cca-2673-488d-83f3-f198be21d6bb","Type":"ContainerDied","Data":"7b1ba1412bbb5919884c0683cfdd780f1eb59964a3ec545d2d9ab4cdd35c3026"}
Sep 11 18:44:36.665310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.665249    2582 scope.go:117] "RemoveContainer" containerID="8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"
Sep 11 18:44:36.673905 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.673883    2582 scope.go:117] "RemoveContainer" containerID="64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"
Sep 11 18:44:36.681574 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.681556    2582 scope.go:117] "RemoveContainer" containerID="8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"
Sep 11 18:44:36.681789 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:36.681770    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445\": container with ID starting with 8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445 not found: ID does not exist" containerID="8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"
Sep 11 18:44:36.681860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.681798    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445"} err="failed to get container status \"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445\": rpc error: code = NotFound desc = could not find container \"8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445\": container with ID starting with 8ffc1e0d82cfbc8a9dd5e044d5c599be03f62f870fae1d0aadeea38c7b91a445 not found: ID does not exist"
Sep 11 18:44:36.681860 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.681821    2582 scope.go:117] "RemoveContainer" containerID="64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"
Sep 11 18:44:36.681935 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.681870    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:44:36.682072 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:44:36.682056    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc\": container with ID starting with 64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc not found: ID does not exist" containerID="64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"
Sep 11 18:44:36.682121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.682084    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc"} err="failed to get container status \"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc\": rpc error: code = NotFound desc = could not find container \"64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc\": container with ID starting with 64306c4b0fa724d2d760f33b7863a9a3ab41c5166c93544a87ca01eab2aaaadc not found: ID does not exist"
Sep 11 18:44:36.685481 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:36.685461    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c8b274-openshift-default-69b49d4d6b-5cs2f"]
Sep 11 18:44:38.553772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:38.553738    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" path="/var/lib/kubelet/pods/86aa2cca-2673-488d-83f3-f198be21d6bb/volumes"
Sep 11 18:44:51.253129 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:44:51.253088    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:45:00.136151 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136118    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"]
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136439    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="203b723e-af7f-4949-a05e-53a5a99d7fc3" containerName="payload-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136452    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="203b723e-af7f-4949-a05e-53a5a99d7fc3" containerName="payload-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136466    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136472    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136481    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136487    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136496    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" containerName="revoke-keys"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136501    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" containerName="revoke-keys"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136507    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b0d8351-e77f-4566-854b-30897f10f78f" containerName="payload-pre-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136511    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b0d8351-e77f-4566-854b-30897f10f78f" containerName="payload-pre-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136519    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" containerName="payload-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136524    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" containerName="payload-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136532    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428aa947-530b-4eb7-84db-b39697612f30" containerName="payload-pre-processing"
Sep 11 18:45:00.136545 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136537    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="428aa947-530b-4eb7-84db-b39697612f30" containerName="payload-pre-processing"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136584    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="00b6ce23-3bcb-4e59-a29e-8dc9c55ceb5e" containerName="payload-processing"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136591    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b0d8351-e77f-4566-854b-30897f10f78f" containerName="payload-pre-processing"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136597    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136603    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="203b723e-af7f-4949-a05e-53a5a99d7fc3" containerName="payload-processing"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136610    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="428aa947-530b-4eb7-84db-b39697612f30" containerName="payload-pre-processing"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136616    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="86aa2cca-2673-488d-83f3-f198be21d6bb" containerName="istio-proxy"
Sep 11 18:45:00.136964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.136622    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e107ed4-3fb8-40ae-949d-1db3a0af1d6f" containerName="revoke-keys"
Sep 11 18:45:00.140898 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.140883    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"
Sep 11 18:45:00.143199 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.143164    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:45:00.150348 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.150326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"
Sep 11 18:45:00.154331 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.154305    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"]
Sep 11 18:45:00.422155 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.422042    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"]
Sep 11 18:45:00.424445 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:45:00.424414    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76b9662d_7dc0_44a1_9786_90418bf972ec.slice/crio-13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b WatchSource:0}: Error finding container 13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b: Status 404 returned error can't find the container with id 13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b
Sep 11 18:45:00.774916 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.774818    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm" event={"ID":"76b9662d-7dc0-44a1-9786-90418bf972ec","Type":"ContainerStarted","Data":"e0815de9ad59b0a20764549b73fdb5377d0fa985b4f68ffdc9a63a801e2dc599"}
Sep 11 18:45:00.774916 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.774878    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm" event={"ID":"76b9662d-7dc0-44a1-9786-90418bf972ec","Type":"ContainerStarted","Data":"13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b"}
Sep 11 18:45:00.790896 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:00.790837    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm" podStartSLOduration=0.790822669 podStartE2EDuration="790.822669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:00.789589037 +0000 UTC m=+1770.675561527" watchObservedRunningTime="2026-09-11 18:45:00.790822669 +0000 UTC m=+1770.676795159"
Sep 11 18:45:01.780542 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:01.780507    2582 generic.go:356] "Generic (PLEG): container finished" podID="76b9662d-7dc0-44a1-9786-90418bf972ec" containerID="e0815de9ad59b0a20764549b73fdb5377d0fa985b4f68ffdc9a63a801e2dc599" exitCode=0
Sep 11 18:45:01.780907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:01.780595    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm" event={"ID":"76b9662d-7dc0-44a1-9786-90418bf972ec","Type":"ContainerDied","Data":"e0815de9ad59b0a20764549b73fdb5377d0fa985b4f68ffdc9a63a801e2dc599"}
Sep 11 18:45:02.912085 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:02.912060    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"
Sep 11 18:45:03.791465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:03.791438    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm"
Sep 11 18:45:03.791465 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:03.791439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819205-7krzm" event={"ID":"76b9662d-7dc0-44a1-9786-90418bf972ec","Type":"ContainerDied","Data":"13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b"}
Sep 11 18:45:03.791704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:03.791493    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13801f938be8092c230672dd1578e5eb9959c2165b6d163eace897bdebafb43b"
Sep 11 18:45:06.271569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.271528    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" podUID="a3384e3d-6987-4010-839b-50209fe1b383" containerName="main" containerID="cri-o://493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865" gracePeriod=45
Sep 11 18:45:06.511868 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.511838    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:45:06.580641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580569    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580604    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmzxs\" (UniqueName: \"kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580641 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580644    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580669    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs\") pod \"a3384e3d-6987-4010-839b-50209fe1b383\" (UID: \"a3384e3d-6987-4010-839b-50209fe1b383\") "
Sep 11 18:45:06.580972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580887    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache" (OuterVolumeSpecName: "model-cache") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:06.580972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580923    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:06.580972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.580966    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home" (OuterVolumeSpecName: "home") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:06.581217 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.581015    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.581217 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.581034    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.582967 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.582938    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:45:06.583148 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.583128    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs" (OuterVolumeSpecName: "kube-api-access-fmzxs") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "kube-api-access-fmzxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:45:06.583243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.583130    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm" (OuterVolumeSpecName: "dshm") pod "a3384e3d-6987-4010-839b-50209fe1b383" (UID: "a3384e3d-6987-4010-839b-50209fe1b383"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:45:06.681968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.681937    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3384e3d-6987-4010-839b-50209fe1b383-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.681968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.681962    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmzxs\" (UniqueName: \"kubernetes.io/projected/a3384e3d-6987-4010-839b-50209fe1b383-kube-api-access-fmzxs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.681968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.681972    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.682211 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.681980    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a3384e3d-6987-4010-839b-50209fe1b383-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:45:06.807528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.807497    2582 generic.go:356] "Generic (PLEG): container finished" podID="a3384e3d-6987-4010-839b-50209fe1b383" containerID="493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865" exitCode=0
Sep 11 18:45:06.807688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.807542    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"
Sep 11 18:45:06.807688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.807585    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" event={"ID":"a3384e3d-6987-4010-839b-50209fe1b383","Type":"ContainerDied","Data":"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"}
Sep 11 18:45:06.807688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.807635    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7" event={"ID":"a3384e3d-6987-4010-839b-50209fe1b383","Type":"ContainerDied","Data":"3bf60a3828191475db5308c2f2c725b40e9724c027c93f5a2036314fdcaf02ec"}
Sep 11 18:45:06.807688 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.807658    2582 scope.go:117] "RemoveContainer" containerID="493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"
Sep 11 18:45:06.816272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.816256    2582 scope.go:117] "RemoveContainer" containerID="493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"
Sep 11 18:45:06.816529 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:45:06.816512    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865\": container with ID starting with 493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865 not found: ID does not exist" containerID="493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"
Sep 11 18:45:06.816615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.816538    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865"} err="failed to get container status \"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865\": rpc error: code = NotFound desc = could not find container \"493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865\": container with ID starting with 493335ad5e9435d325bfbaa163147f4976b9276ecbe2690ee613c1abef8d6865 not found: ID does not exist"
Sep 11 18:45:06.827704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.827683    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:45:06.833276 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:06.833228    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-ec9107/auth-test-model-ec9107-kserve-78f7f4968c-rwms7"]
Sep 11 18:45:08.552524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:08.552489    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3384e3d-6987-4010-839b-50209fe1b383" path="/var/lib/kubelet/pods/a3384e3d-6987-4010-839b-50209fe1b383/volumes"
Sep 11 18:45:38.458060 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458022    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458373    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3384e3d-6987-4010-839b-50209fe1b383" containerName="main"
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458385    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3384e3d-6987-4010-839b-50209fe1b383" containerName="main"
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458397    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b9662d-7dc0-44a1-9786-90418bf972ec" containerName="cleanup"
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458402    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b9662d-7dc0-44a1-9786-90418bf972ec" containerName="cleanup"
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458448    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b9662d-7dc0-44a1-9786-90418bf972ec" containerName="cleanup"
Sep 11 18:45:38.458531 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.458458    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3384e3d-6987-4010-839b-50209fe1b383" containerName="main"
Sep 11 18:45:38.461275 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.461253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.464089 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.464069    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"sub-test-model-d6c4cc-kserve-self-signed-certs\""
Sep 11 18:45:38.465006 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.464989    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"openshift-service-ca.crt\""
Sep 11 18:45:38.465211 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.465192    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"kube-root-ca.crt\""
Sep 11 18:45:38.465322 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.465250    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"default-dockercfg-wgdr8\""
Sep 11 18:45:38.472778 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.472759    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:45:38.538806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538779    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.538806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538806    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.538992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538830    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.538992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538872    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.538992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538905    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kmn\" (UniqueName: \"kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.538992 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.538968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639514    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639539    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639562    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kmn\" (UniqueName: \"kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639615    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639918 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639897    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.639982 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.639965    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.640048 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.640005    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.641689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.641673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.641843 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.641826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.647558 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.647535    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kmn\" (UniqueName: \"kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn\") pod \"sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:38.770190 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:38.770083    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:39.042750 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:39.042730    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:45:39.044730 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:45:39.044694    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a037429_1b2e_4806_98dc_62fc3cad617e.slice/crio-68b649eed5b63bf875f789e17c2582d128dfee0267271076994e2ade64c1a6bc WatchSource:0}: Error finding container 68b649eed5b63bf875f789e17c2582d128dfee0267271076994e2ade64c1a6bc: Status 404 returned error can't find the container with id 68b649eed5b63bf875f789e17c2582d128dfee0267271076994e2ade64c1a6bc
Sep 11 18:45:39.956401 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:39.956333    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" event={"ID":"6a037429-1b2e-4806-98dc-62fc3cad617e","Type":"ContainerStarted","Data":"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"}
Sep 11 18:45:39.956401 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:39.956409    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" event={"ID":"6a037429-1b2e-4806-98dc-62fc3cad617e","Type":"ContainerStarted","Data":"68b649eed5b63bf875f789e17c2582d128dfee0267271076994e2ade64c1a6bc"}
Sep 11 18:45:39.987400 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:39.987346    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" podStartSLOduration=1.98732595 podStartE2EDuration="1.98732595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:45:39.986150275 +0000 UTC m=+1809.872122764" watchObservedRunningTime="2026-09-11 18:45:39.98732595 +0000 UTC m=+1809.873298440"
Sep 11 18:45:48.771181 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:48.771131    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:48.771689 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:48.771438    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:48.783660 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:48.783634    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:45:49.009811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:45:49.009776    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:46:21.326835 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:21.326801    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:46:36.345313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.345265    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" podUID="6a037429-1b2e-4806-98dc-62fc3cad617e" containerName="main" containerID="cri-o://6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace" gracePeriod=45
Sep 11 18:46:36.591201 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.591156    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:46:36.734708 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734680    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.734874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4kmn\" (UniqueName: \"kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.734874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734767    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.734874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734788    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.734874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.734874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734873    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home\") pod \"6a037429-1b2e-4806-98dc-62fc3cad617e\" (UID: \"6a037429-1b2e-4806-98dc-62fc3cad617e\") "
Sep 11 18:46:36.735126 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.734975    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:36.735126 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.735076    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:36.735267 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.735140    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:36.735267 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.735214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home" (OuterVolumeSpecName: "home") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:36.736996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.736969    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn" (OuterVolumeSpecName: "kube-api-access-l4kmn") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "kube-api-access-l4kmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:46:36.737095 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.736997    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm" (OuterVolumeSpecName: "dshm") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:46:36.737095 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.737005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a037429-1b2e-4806-98dc-62fc3cad617e" (UID: "6a037429-1b2e-4806-98dc-62fc3cad617e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:46:36.835945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.835918    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kmn\" (UniqueName: \"kubernetes.io/projected/6a037429-1b2e-4806-98dc-62fc3cad617e-kube-api-access-l4kmn\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:36.835945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.835940    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:36.835945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.835949    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a037429-1b2e-4806-98dc-62fc3cad617e-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:36.836160 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.835958    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:36.836160 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:36.835965    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a037429-1b2e-4806-98dc-62fc3cad617e-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:46:37.225220 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.225159    2582 generic.go:356] "Generic (PLEG): container finished" podID="6a037429-1b2e-4806-98dc-62fc3cad617e" containerID="6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace" exitCode=0
Sep 11 18:46:37.225453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.225427    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"
Sep 11 18:46:37.225567 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.225544    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" event={"ID":"6a037429-1b2e-4806-98dc-62fc3cad617e","Type":"ContainerDied","Data":"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"}
Sep 11 18:46:37.225672 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.225657    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z" event={"ID":"6a037429-1b2e-4806-98dc-62fc3cad617e","Type":"ContainerDied","Data":"68b649eed5b63bf875f789e17c2582d128dfee0267271076994e2ade64c1a6bc"}
Sep 11 18:46:37.225771 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.225760    2582 scope.go:117] "RemoveContainer" containerID="6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"
Sep 11 18:46:37.238617 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.238596    2582 scope.go:117] "RemoveContainer" containerID="6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"
Sep 11 18:46:37.238882 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:46:37.238860    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace\": container with ID starting with 6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace not found: ID does not exist" containerID="6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"
Sep 11 18:46:37.238950 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.238889    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace"} err="failed to get container status \"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace\": rpc error: code = NotFound desc = could not find container \"6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace\": container with ID starting with 6c555dde1bad6775ecb39c5cee59f41edcbadeeccc693576a153811d64ee8ace not found: ID does not exist"
Sep 11 18:46:37.249324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.249300    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:46:37.252888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:37.252865    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/sub-test-model-d6c4cc-kserve-b49b4db5c-4xs6z"]
Sep 11 18:46:38.553976 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:38.553946    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a037429-1b2e-4806-98dc-62fc3cad617e" path="/var/lib/kubelet/pods/6a037429-1b2e-4806-98dc-62fc3cad617e/volumes"
Sep 11 18:46:52.855159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.855117    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:46:52.855710 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.855691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a037429-1b2e-4806-98dc-62fc3cad617e" containerName="main"
Sep 11 18:46:52.855763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.855714    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a037429-1b2e-4806-98dc-62fc3cad617e" containerName="main"
Sep 11 18:46:52.855803 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.855796    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a037429-1b2e-4806-98dc-62fc3cad617e" containerName="main"
Sep 11 18:46:52.860223 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.860207    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.863012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.862994    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"kube-root-ca.crt\""
Sep 11 18:46:52.863776 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.863753    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"rate-test-model-d6c4cc-kserve-self-signed-certs\""
Sep 11 18:46:52.863888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.863754    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"openshift-service-ca.crt\""
Sep 11 18:46:52.863888 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.863830    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-d6c4cc\"/\"default-dockercfg-wgdr8\""
Sep 11 18:46:52.868327 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.868301    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:46:52.964798 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.964759    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.964968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.964828    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.964968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.964854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpql\" (UniqueName: \"kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.964968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.964873    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.964968 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.964941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:52.965133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:52.965044    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066112    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpql\" (UniqueName: \"kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066276    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066330 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066330    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066562    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066656 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066634    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.066758 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.066660    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.068460 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.068435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.068623 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.068608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.075880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.075857    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpql\" (UniqueName: \"kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql\") pod \"rate-test-model-d6c4cc-kserve-85557cb479-mhg6d\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") " pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.170012 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.169982    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:46:53.442408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:53.442382    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:46:53.443876 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:46:53.443850    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98f903a2_f6e0_4ebb_a0db_f91a5c6a33b0.slice/crio-d4a5844ee0b17fa7086b3c99e9a8a33ca09b26586878e5fcb00cba7a4a7e2c06 WatchSource:0}: Error finding container d4a5844ee0b17fa7086b3c99e9a8a33ca09b26586878e5fcb00cba7a4a7e2c06: Status 404 returned error can't find the container with id d4a5844ee0b17fa7086b3c99e9a8a33ca09b26586878e5fcb00cba7a4a7e2c06
Sep 11 18:46:54.307603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:54.307558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" event={"ID":"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0","Type":"ContainerStarted","Data":"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"}
Sep 11 18:46:54.307964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:54.307616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" event={"ID":"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0","Type":"ContainerStarted","Data":"d4a5844ee0b17fa7086b3c99e9a8a33ca09b26586878e5fcb00cba7a4a7e2c06"}
Sep 11 18:46:54.326448 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:46:54.326402    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" podStartSLOduration=2.32638659 podStartE2EDuration="2.32638659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:46:54.325532151 +0000 UTC m=+1884.211504640" watchObservedRunningTime="2026-09-11 18:46:54.32638659 +0000 UTC m=+1884.212359078"
Sep 11 18:47:03.170379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:03.170344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:03.170780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:03.170397    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:03.182794 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:03.182773    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:03.360064 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:03.360034    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:35.783455 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:35.783414    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:47:36.507247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.507213    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:47:36.510441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.510422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.518318 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.518297    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-461acb-openshift-default-dockercfg-7n8pv\""
Sep 11 18:47:36.542851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.542824    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:47:36.641946 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.641915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.641952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjr7\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.641979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642048    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642069    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642144 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642196    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642255    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.642438 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.642289    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743509 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743466    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743535    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743571    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743597    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743649    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjr7\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.743713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743710    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744025 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743929    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744025 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743943    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744025 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744188 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.743956    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744188 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.744085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744264 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.744237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.744622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.744602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.745857 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.745835    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.746261 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.746240    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.753572 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.753554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjr7\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.753678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.753658    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token\") pod \"e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") " pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:36.820490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:36.820403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:37.097459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.097378    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:47:37.104804 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:47:37.104776    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6deb4fc4_81fd_47db_a7e5_a8ff1260968e.slice/crio-c92db7daedafaf35dffd619cbe9298bdaaa1ac7c6589a37174b609e2d675717a WatchSource:0}: Error finding container c92db7daedafaf35dffd619cbe9298bdaaa1ac7c6589a37174b609e2d675717a: Status 404 returned error can't find the container with id c92db7daedafaf35dffd619cbe9298bdaaa1ac7c6589a37174b609e2d675717a
Sep 11 18:47:37.107037 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.107003    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:37.107110 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.107083    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:37.107152 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.107120    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:37.505032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.504994    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerStarted","Data":"f5a248e59bf059be4efe39b33124cc2e5ab6dbee7bf8483bf773d3da54abc227"}
Sep 11 18:47:37.505221 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.505054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerStarted","Data":"c92db7daedafaf35dffd619cbe9298bdaaa1ac7c6589a37174b609e2d675717a"}
Sep 11 18:47:37.526257 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.526205    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podStartSLOduration=1.526189716 podStartE2EDuration="1.526189716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:37.524612152 +0000 UTC m=+1927.410584637" watchObservedRunningTime="2026-09-11 18:47:37.526189716 +0000 UTC m=+1927.412162195"
Sep 11 18:47:37.821116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:37.821028    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:38.832586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:38.832555    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:39.515682 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:39.515640    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:39.516634 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:39.516617    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:41.231313 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.231281    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:47:41.234579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.234554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.237388 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.237364    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-461acb-serving-cert\""
Sep 11 18:47:41.245409 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.245386    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:47:41.385843 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.385807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.385843 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.385848    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.386054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.385976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqjj4\" (UniqueName: \"kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.386054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.386016    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.487142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.487036    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqjj4\" (UniqueName: \"kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.487142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.487096    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.487142 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.487128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.487466 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.487151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.487466 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:47:41.487266    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-serving-cert: secret "maas-api-e2e-ipp-a-461acb-serving-cert" not found
Sep 11 18:47:41.487466 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:47:41.487352    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls podName:c328aac6-50c8-4f0b-9883-3dd7c450b580 nodeName:}" failed. No retries permitted until 2026-09-11 18:47:41.987329255 +0000 UTC m=+1931.873301740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls") pod "maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" (UID: "c328aac6-50c8-4f0b-9883-3dd7c450b580") : secret "maas-api-e2e-ipp-a-461acb-serving-cert" not found
Sep 11 18:47:41.487921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.487901    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.489629 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.489603    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.497218 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.497190    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqjj4\" (UniqueName: \"kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.882115 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.882035    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:47:41.885368 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.885350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:41.889635 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.889609    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-461acb-dockercfg-ggf7k\""
Sep 11 18:47:41.889777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.889760    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-461acb\""
Sep 11 18:47:41.914468 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.914443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:47:41.921453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.921429    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:47:41.924564 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.924549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:41.958586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.958552    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:47:41.991950 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.991911    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8s57\" (UniqueName: \"kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:41.992245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.992219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:41.992441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.992413    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:41.995882 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:41.995856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") pod \"maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:42.092987 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.092941    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj9g\" (UniqueName: \"kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.093200 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.092996    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8s57\" (UniqueName: \"kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:42.093200 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.093027    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:42.093200 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.093077    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.093770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.093750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:42.101482 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.101457    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8s57\" (UniqueName: \"kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57\") pod \"payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:42.146991 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.146935    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:42.193700 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.193665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.193923 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.193754    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj9g\" (UniqueName: \"kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.195676 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.194977    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.197289 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.197037    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:42.202627 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.202600    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj9g\" (UniqueName: \"kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g\") pod \"payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.234782 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.234751    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:42.419404 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.419379    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:47:42.422047 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:47:42.422026    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc328aac6_50c8_4f0b_9883_3dd7c450b580.slice/crio-5d330c55a0dc01313bfb12bf708b04e08a5cd3e86525d2ac968b1f87d945b28d WatchSource:0}: Error finding container 5d330c55a0dc01313bfb12bf708b04e08a5cd3e86525d2ac968b1f87d945b28d: Status 404 returned error can't find the container with id 5d330c55a0dc01313bfb12bf708b04e08a5cd3e86525d2ac968b1f87d945b28d
Sep 11 18:47:42.423371 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.423349    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:47:42.473640 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.473575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:47:42.475670 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:47:42.475643    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa49128_a74e_4519_8686_2be01aa04df1.slice/crio-76f92bd1e7f212878c27b6640a60b791e6b5d8501acd461098011429e5bb2336 WatchSource:0}: Error finding container 76f92bd1e7f212878c27b6640a60b791e6b5d8501acd461098011429e5bb2336: Status 404 returned error can't find the container with id 76f92bd1e7f212878c27b6640a60b791e6b5d8501acd461098011429e5bb2336
Sep 11 18:47:42.518230 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.518201    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:47:42.525240 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:47:42.525213    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad306d4_914a_4ffa_abcb_f6dc7258d297.slice/crio-c312db0c7e251d5f92a560f06dd3ccf8b560ee8186fe91a1bc3a0f58fd7faa11 WatchSource:0}: Error finding container c312db0c7e251d5f92a560f06dd3ccf8b560ee8186fe91a1bc3a0f58fd7faa11: Status 404 returned error can't find the container with id c312db0c7e251d5f92a560f06dd3ccf8b560ee8186fe91a1bc3a0f58fd7faa11
Sep 11 18:47:42.533694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.533661    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" event={"ID":"9aa49128-a74e-4519-8686-2be01aa04df1","Type":"ContainerStarted","Data":"76f92bd1e7f212878c27b6640a60b791e6b5d8501acd461098011429e5bb2336"}
Sep 11 18:47:42.538024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:42.537958    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" event={"ID":"c328aac6-50c8-4f0b-9883-3dd7c450b580","Type":"ContainerStarted","Data":"5d330c55a0dc01313bfb12bf708b04e08a5cd3e86525d2ac968b1f87d945b28d"}
Sep 11 18:47:43.547814 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.546814    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" event={"ID":"c328aac6-50c8-4f0b-9883-3dd7c450b580","Type":"ContainerStarted","Data":"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"}
Sep 11 18:47:43.548312 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.547969    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:43.550506 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.550476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" event={"ID":"1ad306d4-914a-4ffa-abcb-f6dc7258d297","Type":"ContainerStarted","Data":"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"}
Sep 11 18:47:43.550612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.550531    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:43.550612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.550555    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" event={"ID":"1ad306d4-914a-4ffa-abcb-f6dc7258d297","Type":"ContainerStarted","Data":"c312db0c7e251d5f92a560f06dd3ccf8b560ee8186fe91a1bc3a0f58fd7faa11"}
Sep 11 18:47:43.552052 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.552028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" event={"ID":"9aa49128-a74e-4519-8686-2be01aa04df1","Type":"ContainerStarted","Data":"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"}
Sep 11 18:47:43.552189 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.552159    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:43.569806 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.569760    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" podStartSLOduration=2.001972378 podStartE2EDuration="2.569746872s" totalImagesPullingTime="567.774494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:43.567849135 +0000 UTC m=+1933.453821625" watchObservedRunningTime="2026-09-11 18:47:43.569746872 +0000 UTC m=+1933.455719360"
Sep 11 18:47:43.585636 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.585598    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" podStartSLOduration=2.585585645 podStartE2EDuration="2.585585645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:43.584082541 +0000 UTC m=+1933.470055030" watchObservedRunningTime="2026-09-11 18:47:43.585585645 +0000 UTC m=+1933.471558136"
Sep 11 18:47:43.600763 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:43.600718    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" podStartSLOduration=2.600703276 podStartE2EDuration="2.600703276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:47:43.598884791 +0000 UTC m=+1933.484857282" watchObservedRunningTime="2026-09-11 18:47:43.600703276 +0000 UTC m=+1933.486675796"
Sep 11 18:47:50.566746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:50.566718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:47:50.812660 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:50.812603    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" podUID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" containerName="main" containerID="cri-o://bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d" gracePeriod=45
Sep 11 18:47:51.072392 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.072329    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:51.173002 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.172956    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173154 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173154 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173069    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173154 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173092    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173154 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173148    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmpql\" (UniqueName: \"kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173243    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs\") pod \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\" (UID: \"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0\") "
Sep 11 18:47:51.173324 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173245    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home" (OuterVolumeSpecName: "home") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:51.173390 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:51.173504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173480    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.173504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173498    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.173646 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.173626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache" (OuterVolumeSpecName: "model-cache") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:51.175121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.175100    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm" (OuterVolumeSpecName: "dshm") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:47:51.175311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.175292    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:47:51.175379 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.175298    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql" (OuterVolumeSpecName: "kube-api-access-rmpql") pod "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" (UID: "98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0"). InnerVolumeSpecName "kube-api-access-rmpql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:47:51.274384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.274352    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.274384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.274379    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.274384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.274387    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.274384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.274395    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmpql\" (UniqueName: \"kubernetes.io/projected/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0-kube-api-access-rmpql\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:47:51.589624 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.589579    2582 generic.go:356] "Generic (PLEG): container finished" podID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" containerID="bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d" exitCode=0
Sep 11 18:47:51.590108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.589645    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" event={"ID":"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0","Type":"ContainerDied","Data":"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"}
Sep 11 18:47:51.590108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.589680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d" event={"ID":"98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0","Type":"ContainerDied","Data":"d4a5844ee0b17fa7086b3c99e9a8a33ca09b26586878e5fcb00cba7a4a7e2c06"}
Sep 11 18:47:51.590108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.589701    2582 scope.go:117] "RemoveContainer" containerID="bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"
Sep 11 18:47:51.590108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.589857    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"
Sep 11 18:47:51.608714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.608692    2582 scope.go:117] "RemoveContainer" containerID="bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"
Sep 11 18:47:51.609041 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:47:51.609021    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d\": container with ID starting with bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d not found: ID does not exist" containerID="bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"
Sep 11 18:47:51.609116 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.609049    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d"} err="failed to get container status \"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d\": rpc error: code = NotFound desc = could not find container \"bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d\": container with ID starting with bbe33c277ea88e9f87622fe6c2c6380fa1c97b3e94a36c7d333cf180c2e4ef7d not found: ID does not exist"
Sep 11 18:47:51.618284 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.618259    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:47:51.622649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:51.622627    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-d6c4cc/rate-test-model-d6c4cc-kserve-85557cb479-mhg6d"]
Sep 11 18:47:52.554511 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:52.554472    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" path="/var/lib/kubelet/pods/98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0/volumes"
Sep 11 18:47:54.558305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:54.558278    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:47:54.558659 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:54.558340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:47:54.724880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:54.724842    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": read tcp 10.132.0.2:56236->10.132.0.73:15021: read: connection reset by peer" start-of-body=
Sep 11 18:47:54.725069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:54.724917    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": read tcp 10.132.0.2:56236->10.132.0.73:15021: read: connection reset by peer"
Sep 11 18:47:55.615259 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.615225    2582 generic.go:356] "Generic (PLEG): container finished" podID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerID="f5a248e59bf059be4efe39b33124cc2e5ab6dbee7bf8483bf773d3da54abc227" exitCode=0
Sep 11 18:47:55.615639 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.615295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerDied","Data":"f5a248e59bf059be4efe39b33124cc2e5ab6dbee7bf8483bf773d3da54abc227"}
Sep 11 18:47:55.615714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.615652    2582 scope.go:117] "RemoveContainer" containerID="f5a248e59bf059be4efe39b33124cc2e5ab6dbee7bf8483bf773d3da54abc227"
Sep 11 18:47:55.616744 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.616713    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:55.616817 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.616800    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:55.616857 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:55.616832    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 11 18:47:56.621678 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:56.621647    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerStarted","Data":"02753ee0e3e1b1b5ebc3bda3e2c021898d171f3a72e3229b450b4531ae08eec2"}
Sep 11 18:47:56.820747 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:56.820705    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:47:56.820931 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:56.820909    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 18:47:56.820979 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:56.820965    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 18:47:57.821772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:57.821733    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 18:47:57.822216 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:57.821792    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 18:47:58.821223 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:58.821157    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 18:47:58.821425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:58.821259    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 18:47:59.821094 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:59.821060    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 18:47:59.821518 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:47:59.821122    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 18:48:00.820855 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:00.820815    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Sep 11 18:48:00.821034 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:00.820870    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Sep 11 18:48:01.824632 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:01.824595    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:48:01.825159 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:01.824858    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:48:01.825484 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:01.825466    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:48:12.020017 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.019980    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:48:12.021121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.021083    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" containerName="main"
Sep 11 18:48:12.021121 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.021121    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" containerName="main"
Sep 11 18:48:12.021384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.021369    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f903a2-f6e0-4ebb-a0db-f91a5c6a33b0" containerName="main"
Sep 11 18:48:12.027972 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.027950    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.030737 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.030714    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b01b33\""
Sep 11 18:48:12.031022 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.030998    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b01b33-dockercfg-xmklk\""
Sep 11 18:48:12.033286 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.033264    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:48:12.036540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.036523    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.039467 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.039446    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:48:12.049419 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.049399    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:48:12.155451 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.155418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s65vz\" (UniqueName: \"kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.155615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.155457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.155615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.155485    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.155615 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.155532    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x24f\" (UniqueName: \"kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.256101 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256071    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s65vz\" (UniqueName: \"kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.256277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256106    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.256277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.256277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256152    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x24f\" (UniqueName: \"kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.256875 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.256914 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.256883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.264893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.264863    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x24f\" (UniqueName: \"kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f\") pod \"payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.265029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.264926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s65vz\" (UniqueName: \"kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz\") pod \"payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.339086 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.339024    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.348277 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.348253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.618851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.618828    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:48:12.619866 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:48:12.619844    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb61ba98_345e_42c7_88c0_3993a2927851.slice/crio-0569ceea9339d2920ee62d943bb4ec68755ab0043a051b9ab9952eb98181e3fe WatchSource:0}: Error finding container 0569ceea9339d2920ee62d943bb4ec68755ab0043a051b9ab9952eb98181e3fe: Status 404 returned error can't find the container with id 0569ceea9339d2920ee62d943bb4ec68755ab0043a051b9ab9952eb98181e3fe
Sep 11 18:48:12.632332 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.632310    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:48:12.633119 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:48:12.633100    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7114e69_1de6_4188_9b81_ead77d852c26.slice/crio-94132ccd52d087d1288ce0270ac7459b08f48b415cb88608e1e25604d458fc89 WatchSource:0}: Error finding container 94132ccd52d087d1288ce0270ac7459b08f48b415cb88608e1e25604d458fc89: Status 404 returned error can't find the container with id 94132ccd52d087d1288ce0270ac7459b08f48b415cb88608e1e25604d458fc89
Sep 11 18:48:12.701425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.701297    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" event={"ID":"eb61ba98-345e-42c7-88c0-3993a2927851","Type":"ContainerStarted","Data":"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"}
Sep 11 18:48:12.701425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.701367    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" event={"ID":"eb61ba98-345e-42c7-88c0-3993a2927851","Type":"ContainerStarted","Data":"0569ceea9339d2920ee62d943bb4ec68755ab0043a051b9ab9952eb98181e3fe"}
Sep 11 18:48:12.701425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.701398    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:12.703205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.703149    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" event={"ID":"b7114e69-1de6-4188-9b81-ead77d852c26","Type":"ContainerStarted","Data":"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"}
Sep 11 18:48:12.703345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.703224    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" event={"ID":"b7114e69-1de6-4188-9b81-ead77d852c26","Type":"ContainerStarted","Data":"94132ccd52d087d1288ce0270ac7459b08f48b415cb88608e1e25604d458fc89"}
Sep 11 18:48:12.703345 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.703259    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:12.724587 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.724547    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" podStartSLOduration=1.7245331 podStartE2EDuration="1.7245331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:12.722662635 +0000 UTC m=+1962.608635123" watchObservedRunningTime="2026-09-11 18:48:12.7245331 +0000 UTC m=+1962.610505589"
Sep 11 18:48:12.741646 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:12.741601    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" podStartSLOduration=0.741578525 podStartE2EDuration="741.578525ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:48:12.740959501 +0000 UTC m=+1962.626931992" watchObservedRunningTime="2026-09-11 18:48:12.741578525 +0000 UTC m=+1962.627551014"
Sep 11 18:48:23.711136 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:23.711054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:48:23.711513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:23.711141    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:48:38.404441 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:48:38.404410    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc_b7114e69-1de6-4188-9b81-ead77d852c26/payload-processing/0.log"
Sep 11 18:49:24.414969 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:24.414938    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk_1ad306d4-914a-4ffa-abcb-f6dc7258d297/payload-processing/0.log"
Sep 11 18:49:24.648884 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:24.648858    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc_b7114e69-1de6-4188-9b81-ead77d852c26/payload-processing/0.log"
Sep 11 18:49:30.921464 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.921433    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:49:30.924738 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.924718    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:30.927185 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.927139    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d12501-serving-cert\""
Sep 11 18:49:30.937630 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.937602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:30.937770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.937677    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz9cp\" (UniqueName: \"kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:30.937770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.937718    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:30.937897 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.937785    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:30.938197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:30.938160    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:49:31.039262 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.039219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.039604 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.039312    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz9cp\" (UniqueName: \"kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.039604 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.039360    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.039604 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.039412    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.040852 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.040824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.041151 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:49:31.041133    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-serving-cert: secret "maas-api-e2e-ipp-cleanup-d12501-serving-cert" not found
Sep 11 18:49:31.041302 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:49:31.041287    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls podName:8bb6b7c8-d62d-4414-9a0e-508823abfe69 nodeName:}" failed. No retries permitted until 2026-09-11 18:49:31.541270363 +0000 UTC m=+2041.427242838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" (UID: "8bb6b7c8-d62d-4414-9a0e-508823abfe69") : secret "maas-api-e2e-ipp-cleanup-d12501-serving-cert" not found
Sep 11 18:49:31.042993 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.042972    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.048264 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.048243    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz9cp\" (UniqueName: \"kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.544433 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.544399    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.546704 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.546675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:31.835429 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:31.835349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:32.111058 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:32.111032    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:49:32.113361 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:49:32.113328    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb6b7c8_d62d_4414_9a0e_508823abfe69.slice/crio-59b4789c6b595c82a56d2b266ac9444e90f101bceddc32c3631f092f4115653f WatchSource:0}: Error finding container 59b4789c6b595c82a56d2b266ac9444e90f101bceddc32c3631f092f4115653f: Status 404 returned error can't find the container with id 59b4789c6b595c82a56d2b266ac9444e90f101bceddc32c3631f092f4115653f
Sep 11 18:49:33.087713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:33.087677    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" event={"ID":"8bb6b7c8-d62d-4414-9a0e-508823abfe69","Type":"ContainerStarted","Data":"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"}
Sep 11 18:49:33.087925 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:33.087737    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" event={"ID":"8bb6b7c8-d62d-4414-9a0e-508823abfe69","Type":"ContainerStarted","Data":"59b4789c6b595c82a56d2b266ac9444e90f101bceddc32c3631f092f4115653f"}
Sep 11 18:49:33.087925 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:33.087796    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:33.105793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:33.105737    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" podStartSLOduration=2.696206286 podStartE2EDuration="3.105718756s" totalImagesPullingTime="409.51247ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:49:33.103819806 +0000 UTC m=+2042.989792294" watchObservedRunningTime="2026-09-11 18:49:33.105718756 +0000 UTC m=+2042.991691246"
Sep 11 18:49:39.097838 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:39.097802    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:49:59.848541 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:59.848467    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:49:59.849081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:49:59.848712    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" podUID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" containerName="maas-api" containerID="cri-o://03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5" gracePeriod=30
Sep 11 18:50:00.098301 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.098276    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:50:00.191674 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.191649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gz9cp\" (UniqueName: \"kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp\") pod \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") "
Sep 11 18:50:00.191821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.191745    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") pod \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") "
Sep 11 18:50:00.191821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.191780    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca\") pod \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") "
Sep 11 18:50:00.191821 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.191809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs\") pod \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\" (UID: \"8bb6b7c8-d62d-4414-9a0e-508823abfe69\") "
Sep 11 18:50:00.192186 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.192144    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8bb6b7c8-d62d-4414-9a0e-508823abfe69" (UID: "8bb6b7c8-d62d-4414-9a0e-508823abfe69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:00.193773 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.193744    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8bb6b7c8-d62d-4414-9a0e-508823abfe69" (UID: "8bb6b7c8-d62d-4414-9a0e-508823abfe69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:00.193885 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.193782    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8bb6b7c8-d62d-4414-9a0e-508823abfe69" (UID: "8bb6b7c8-d62d-4414-9a0e-508823abfe69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:00.193885 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.193823    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp" (OuterVolumeSpecName: "kube-api-access-gz9cp") pod "8bb6b7c8-d62d-4414-9a0e-508823abfe69" (UID: "8bb6b7c8-d62d-4414-9a0e-508823abfe69"). InnerVolumeSpecName "kube-api-access-gz9cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:00.222780 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.222753    2582 generic.go:356] "Generic (PLEG): container finished" podID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" containerID="03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5" exitCode=0
Sep 11 18:50:00.222907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.222793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" event={"ID":"8bb6b7c8-d62d-4414-9a0e-508823abfe69","Type":"ContainerDied","Data":"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"}
Sep 11 18:50:00.222907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.222800    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"
Sep 11 18:50:00.222907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.222839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p" event={"ID":"8bb6b7c8-d62d-4414-9a0e-508823abfe69","Type":"ContainerDied","Data":"59b4789c6b595c82a56d2b266ac9444e90f101bceddc32c3631f092f4115653f"}
Sep 11 18:50:00.222907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.222862    2582 scope.go:117] "RemoveContainer" containerID="03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"
Sep 11 18:50:00.237539 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.237519    2582 scope.go:117] "RemoveContainer" containerID="03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"
Sep 11 18:50:00.237858 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:50:00.237838    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5\": container with ID starting with 03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5 not found: ID does not exist" containerID="03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"
Sep 11 18:50:00.237934 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.237874    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5"} err="failed to get container status \"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5\": rpc error: code = NotFound desc = could not find container \"03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5\": container with ID starting with 03472e9f9d33d00560cc52d76f86308848bf3a5395dfc9b0d7c4d86a93911db5 not found: ID does not exist"
Sep 11 18:50:00.248770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.248747    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:50:00.254237 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.254208    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d12501-f49f8445-8cz7p"]
Sep 11 18:50:00.293637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.293608    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.293637 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.293639    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz9cp\" (UniqueName: \"kubernetes.io/projected/8bb6b7c8-d62d-4414-9a0e-508823abfe69-kube-api-access-gz9cp\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.293793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.293653    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb6b7c8-d62d-4414-9a0e-508823abfe69-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.293793 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.293667    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb6b7c8-d62d-4414-9a0e-508823abfe69-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:00.553944 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:00.553863    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" path="/var/lib/kubelet/pods/8bb6b7c8-d62d-4414-9a0e-508823abfe69/volumes"
Sep 11 18:50:25.676649 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.676609    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:50:25.677127 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.676847    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" podUID="c328aac6-50c8-4f0b-9883-3dd7c450b580" containerName="maas-api" containerID="cri-o://739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499" gracePeriod=30
Sep 11 18:50:25.730695 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.730657    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:50:25.731234 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.731197    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" podUID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" containerName="payload-processing" containerID="cri-o://dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1" gracePeriod=30
Sep 11 18:50:25.738330 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:50:25.738298    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc328aac6_50c8_4f0b_9883_3dd7c450b580.slice/crio-conmon-739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499.scope\": RecentStats: unable to find data in memory cache]"
Sep 11 18:50:25.738788 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.738750    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:50:25.739207 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:25.739153    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" podUID="9aa49128-a74e-4519-8686-2be01aa04df1" containerName="payload-pre-processing" containerID="cri-o://3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626" gracePeriod=30
Sep 11 18:50:26.043063 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.043034    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:50:26.046969 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.046944    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:50:26.054471 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.054449    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:50:26.109459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109429    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume\") pod \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109477    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs\") pod \"c328aac6-50c8-4f0b-9883-3dd7c450b580\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109521    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8s57\" (UniqueName: \"kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57\") pod \"9aa49128-a74e-4519-8686-2be01aa04df1\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109545    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqjj4\" (UniqueName: \"kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4\") pod \"c328aac6-50c8-4f0b-9883-3dd7c450b580\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109561    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume\") pod \"9aa49128-a74e-4519-8686-2be01aa04df1\" (UID: \"9aa49128-a74e-4519-8686-2be01aa04df1\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsj9g\" (UniqueName: \"kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g\") pod \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\" (UID: \"1ad306d4-914a-4ffa-abcb-f6dc7258d297\") "
Sep 11 18:50:26.109638 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109628    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca\") pod \"c328aac6-50c8-4f0b-9883-3dd7c450b580\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") "
Sep 11 18:50:26.109970 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109682    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") pod \"c328aac6-50c8-4f0b-9883-3dd7c450b580\" (UID: \"c328aac6-50c8-4f0b-9883-3dd7c450b580\") "
Sep 11 18:50:26.109970 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109892    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ad306d4-914a-4ffa-abcb-f6dc7258d297" (UID: "1ad306d4-914a-4ffa-abcb-f6dc7258d297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:26.109970 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.109938    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9aa49128-a74e-4519-8686-2be01aa04df1" (UID: "9aa49128-a74e-4519-8686-2be01aa04df1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:26.110360 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.110319    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c328aac6-50c8-4f0b-9883-3dd7c450b580" (UID: "c328aac6-50c8-4f0b-9883-3dd7c450b580"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:50:26.111996 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.111968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57" (OuterVolumeSpecName: "kube-api-access-d8s57") pod "9aa49128-a74e-4519-8686-2be01aa04df1" (UID: "9aa49128-a74e-4519-8686-2be01aa04df1"). InnerVolumeSpecName "kube-api-access-d8s57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:26.112088 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.112041    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c328aac6-50c8-4f0b-9883-3dd7c450b580" (UID: "c328aac6-50c8-4f0b-9883-3dd7c450b580"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:26.112236 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.112214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c328aac6-50c8-4f0b-9883-3dd7c450b580" (UID: "c328aac6-50c8-4f0b-9883-3dd7c450b580"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:50:26.112308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.112243    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g" (OuterVolumeSpecName: "kube-api-access-vsj9g") pod "1ad306d4-914a-4ffa-abcb-f6dc7258d297" (UID: "1ad306d4-914a-4ffa-abcb-f6dc7258d297"). InnerVolumeSpecName "kube-api-access-vsj9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:26.112308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.112277    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4" (OuterVolumeSpecName: "kube-api-access-lqjj4") pod "c328aac6-50c8-4f0b-9883-3dd7c450b580" (UID: "c328aac6-50c8-4f0b-9883-3dd7c450b580"). InnerVolumeSpecName "kube-api-access-lqjj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210684    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210711    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8s57\" (UniqueName: \"kubernetes.io/projected/9aa49128-a74e-4519-8686-2be01aa04df1-kube-api-access-d8s57\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210721    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqjj4\" (UniqueName: \"kubernetes.io/projected/c328aac6-50c8-4f0b-9883-3dd7c450b580-kube-api-access-lqjj4\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210731    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa49128-a74e-4519-8686-2be01aa04df1-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210740    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsj9g\" (UniqueName: \"kubernetes.io/projected/1ad306d4-914a-4ffa-abcb-f6dc7258d297-kube-api-access-vsj9g\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210748    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c328aac6-50c8-4f0b-9883-3dd7c450b580-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210757    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c328aac6-50c8-4f0b-9883-3dd7c450b580-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.210769 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.210765    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ad306d4-914a-4ffa-abcb-f6dc7258d297-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:50:26.352956 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.352917    2582 generic.go:356] "Generic (PLEG): container finished" podID="c328aac6-50c8-4f0b-9883-3dd7c450b580" containerID="739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499" exitCode=0
Sep 11 18:50:26.353123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.352966    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"
Sep 11 18:50:26.353123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.353006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" event={"ID":"c328aac6-50c8-4f0b-9883-3dd7c450b580","Type":"ContainerDied","Data":"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"}
Sep 11 18:50:26.353123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.353062    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9" event={"ID":"c328aac6-50c8-4f0b-9883-3dd7c450b580","Type":"ContainerDied","Data":"5d330c55a0dc01313bfb12bf708b04e08a5cd3e86525d2ac968b1f87d945b28d"}
Sep 11 18:50:26.353123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.353087    2582 scope.go:117] "RemoveContainer" containerID="739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"
Sep 11 18:50:26.354947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.354800    2582 generic.go:356] "Generic (PLEG): container finished" podID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" containerID="dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1" exitCode=0
Sep 11 18:50:26.354947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.354833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" event={"ID":"1ad306d4-914a-4ffa-abcb-f6dc7258d297","Type":"ContainerDied","Data":"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"}
Sep 11 18:50:26.354947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.354862    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"
Sep 11 18:50:26.354947 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.354893    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk" event={"ID":"1ad306d4-914a-4ffa-abcb-f6dc7258d297","Type":"ContainerDied","Data":"c312db0c7e251d5f92a560f06dd3ccf8b560ee8186fe91a1bc3a0f58fd7faa11"}
Sep 11 18:50:26.356548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.356522    2582 generic.go:356] "Generic (PLEG): container finished" podID="9aa49128-a74e-4519-8686-2be01aa04df1" containerID="3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626" exitCode=0
Sep 11 18:50:26.356720 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.356572    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"
Sep 11 18:50:26.356720 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.356591    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" event={"ID":"9aa49128-a74e-4519-8686-2be01aa04df1","Type":"ContainerDied","Data":"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"}
Sep 11 18:50:26.356720 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.356636    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl" event={"ID":"9aa49128-a74e-4519-8686-2be01aa04df1","Type":"ContainerDied","Data":"76f92bd1e7f212878c27b6640a60b791e6b5d8501acd461098011429e5bb2336"}
Sep 11 18:50:26.364243 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.364151    2582 scope.go:117] "RemoveContainer" containerID="739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"
Sep 11 18:50:26.364826 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:50:26.364504    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499\": container with ID starting with 739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499 not found: ID does not exist" containerID="739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"
Sep 11 18:50:26.364826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.364539    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499"} err="failed to get container status \"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499\": rpc error: code = NotFound desc = could not find container \"739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499\": container with ID starting with 739fa7f5645dfb4f0b6dd1b658543591c9a517f5cccaebdd4a37138661792499 not found: ID does not exist"
Sep 11 18:50:26.364826 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.364560    2582 scope.go:117] "RemoveContainer" containerID="dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"
Sep 11 18:50:26.372957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.372940    2582 scope.go:117] "RemoveContainer" containerID="dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"
Sep 11 18:50:26.373244 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:50:26.373226    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1\": container with ID starting with dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1 not found: ID does not exist" containerID="dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"
Sep 11 18:50:26.373310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.373251    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1"} err="failed to get container status \"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1\": rpc error: code = NotFound desc = could not find container \"dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1\": container with ID starting with dacdb286d2867e2200d161dc58316a6baf7775f3522d1673b4ef2f8aad7009a1 not found: ID does not exist"
Sep 11 18:50:26.373310 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.373266    2582 scope.go:117] "RemoveContainer" containerID="3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"
Sep 11 18:50:26.380484 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.380465    2582 scope.go:117] "RemoveContainer" containerID="3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"
Sep 11 18:50:26.380732 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:50:26.380714    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626\": container with ID starting with 3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626 not found: ID does not exist" containerID="3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"
Sep 11 18:50:26.380778 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.380739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626"} err="failed to get container status \"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626\": rpc error: code = NotFound desc = could not find container \"3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626\": container with ID starting with 3e05780c3af5e80e1c274f3360d98cd8c7db0c6288e6a0dafb463abf78e3b626 not found: ID does not exist"
Sep 11 18:50:26.396375 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.396350    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:50:26.403424 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.403396    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-461acb-85f7ff5d54-g52l9"]
Sep 11 18:50:26.416262 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.416237    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:50:26.420813 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.420793    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-461acb-6d5dbddc8-gggpl"]
Sep 11 18:50:26.433445 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.433419    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:50:26.439569 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.439543    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-461acb-85b9b74ff5-6w5dk"]
Sep 11 18:50:26.553672 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.553583    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" path="/var/lib/kubelet/pods/1ad306d4-914a-4ffa-abcb-f6dc7258d297/volumes"
Sep 11 18:50:26.554014 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.553994    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa49128-a74e-4519-8686-2be01aa04df1" path="/var/lib/kubelet/pods/9aa49128-a74e-4519-8686-2be01aa04df1/volumes"
Sep 11 18:50:26.554423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:26.554405    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c328aac6-50c8-4f0b-9883-3dd7c450b580" path="/var/lib/kubelet/pods/c328aac6-50c8-4f0b-9883-3dd7c450b580/volumes"
Sep 11 18:50:30.879783 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:30.879754    2582 scope.go:117] "RemoveContainer" containerID="fd9326974131961df2d528ba6e9bd15c53ac5c209aa021c9a9d7dee635159abc"
Sep 11 18:50:30.887506 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:50:30.887488    2582 scope.go:117] "RemoveContainer" containerID="7cfe8a0e3328268bf8a40eac712987ad1ee443fd3ff2580af09eb14eba141b93"
Sep 11 18:51:14.494272 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.494234    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:51:14.494755 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.494524    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" containerID="cri-o://02753ee0e3e1b1b5ebc3bda3e2c021898d171f3a72e3229b450b4531ae08eec2" gracePeriod=30
Sep 11 18:51:14.885998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.885923    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"]
Sep 11 18:51:14.886303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886279    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa49128-a74e-4519-8686-2be01aa04df1" containerName="payload-pre-processing"
Sep 11 18:51:14.886303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886300    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa49128-a74e-4519-8686-2be01aa04df1" containerName="payload-pre-processing"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886318    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c328aac6-50c8-4f0b-9883-3dd7c450b580" containerName="maas-api"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886327    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c328aac6-50c8-4f0b-9883-3dd7c450b580" containerName="maas-api"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886360    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" containerName="payload-processing"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886369    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" containerName="payload-processing"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886379    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" containerName="maas-api"
Sep 11 18:51:14.886458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886387    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" containerName="maas-api"
Sep 11 18:51:14.886645 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886465    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bb6b7c8-d62d-4414-9a0e-508823abfe69" containerName="maas-api"
Sep 11 18:51:14.886645 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886477    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ad306d4-914a-4ffa-abcb-f6dc7258d297" containerName="payload-processing"
Sep 11 18:51:14.886645 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886486    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa49128-a74e-4519-8686-2be01aa04df1" containerName="payload-pre-processing"
Sep 11 18:51:14.886645 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.886496    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c328aac6-50c8-4f0b-9883-3dd7c450b580" containerName="maas-api"
Sep 11 18:51:14.889721 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.889702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:14.892099 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.892075    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:51:14.897496 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:14.897473    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"]
Sep 11 18:51:15.019162 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.019124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm\" (UID: \"1f2be16e-e8d4-490b-9cf1-c2a73364e05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:15.120537 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.120502    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm\" (UID: \"1f2be16e-e8d4-490b-9cf1-c2a73364e05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:15.121076 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.121057    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm\" (UID: \"1f2be16e-e8d4-490b-9cf1-c2a73364e05a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:15.201948 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.201915    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:15.478941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.478916    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"]
Sep 11 18:51:15.481238 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:51:15.481208    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2be16e_e8d4_490b_9cf1_c2a73364e05a.slice/crio-88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5 WatchSource:0}: Error finding container 88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5: Status 404 returned error can't find the container with id 88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5
Sep 11 18:51:15.596611 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:15.596579    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm" event={"ID":"1f2be16e-e8d4-490b-9cf1-c2a73364e05a","Type":"ContainerStarted","Data":"88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5"}
Sep 11 18:51:16.603290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:16.603198    2582 generic.go:356] "Generic (PLEG): container finished" podID="1f2be16e-e8d4-490b-9cf1-c2a73364e05a" containerID="3b2f8e13d50a659b7faf32778a0d559a2920d0047d14dceb1bbfba08eb8990f8" exitCode=0
Sep 11 18:51:16.603290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:16.603279    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm" event={"ID":"1f2be16e-e8d4-490b-9cf1-c2a73364e05a","Type":"ContainerDied","Data":"3b2f8e13d50a659b7faf32778a0d559a2920d0047d14dceb1bbfba08eb8990f8"}
Sep 11 18:51:16.827365 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:16.827327    2582 patch_prober.go:28] interesting pod/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 11 18:51:16.827519 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:16.827382    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 11 18:51:17.739750 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:17.739727    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:17.842583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:17.842552    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca\") pod \"1f2be16e-e8d4-490b-9cf1-c2a73364e05a\" (UID: \"1f2be16e-e8d4-490b-9cf1-c2a73364e05a\") "
Sep 11 18:51:17.842925 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:17.842907    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f2be16e-e8d4-490b-9cf1-c2a73364e05a" (UID: "1f2be16e-e8d4-490b-9cf1-c2a73364e05a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:17.944029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:17.943996    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f2be16e-e8d4-490b-9cf1-c2a73364e05a-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:18.613321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:18.613291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm" event={"ID":"1f2be16e-e8d4-490b-9cf1-c2a73364e05a","Type":"ContainerDied","Data":"88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5"}
Sep 11 18:51:18.613321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:18.613309    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"
Sep 11 18:51:18.613321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:18.613336    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88cf5b5439810587bbea12ffa09fc42172d616dfbf153379f75e19c1315366d5"
Sep 11 18:51:19.619997 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.619968    2582 generic.go:356] "Generic (PLEG): container finished" podID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerID="02753ee0e3e1b1b5ebc3bda3e2c021898d171f3a72e3229b450b4531ae08eec2" exitCode=0
Sep 11 18:51:19.620362 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.620036    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerDied","Data":"02753ee0e3e1b1b5ebc3bda3e2c021898d171f3a72e3229b450b4531ae08eec2"}
Sep 11 18:51:19.620362 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.620099    2582 scope.go:117] "RemoveContainer" containerID="f5a248e59bf059be4efe39b33124cc2e5ab6dbee7bf8483bf773d3da54abc227"
Sep 11 18:51:19.738580 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.738556    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:51:19.758590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758535    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758590    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758609    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckjr7\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758649    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758694 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758673    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758702    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758747    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs\") pod \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\" (UID: \"6deb4fc4-81fd-47db-a7e5-a8ff1260968e\") "
Sep 11 18:51:19.758941 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758807    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data" (OuterVolumeSpecName: "istio-data") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:51:19.759146 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.758937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:51:19.759146 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.759008    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-data\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.759290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.759162    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:51:19.759290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.759233    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:19.759290 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.759235    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:51:19.761586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.761492    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7" (OuterVolumeSpecName: "kube-api-access-ckjr7") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "kube-api-access-ckjr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:19.761586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.761538    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:51:19.761586 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.761561    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 11 18:51:19.761824 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.761638    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token" (OuterVolumeSpecName: "istio-token") pod "6deb4fc4-81fd-47db-a7e5-a8ff1260968e" (UID: "6deb4fc4-81fd-47db-a7e5-a8ff1260968e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:19.859811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859776    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-credential-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.859811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859804    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckjr7\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-kube-api-access-ckjr7\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.859811 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859816    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istiod-ca-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.860061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859826    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-podinfo\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.860061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859835    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-envoy\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.860061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859842    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-istio-token\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.860061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859849    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-socket\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:19.860061 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:19.859857    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6deb4fc4-81fd-47db-a7e5-a8ff1260968e-workload-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:20.625534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:20.625504    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"
Sep 11 18:51:20.625933 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:20.625540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr" event={"ID":"6deb4fc4-81fd-47db-a7e5-a8ff1260968e","Type":"ContainerDied","Data":"c92db7daedafaf35dffd619cbe9298bdaaa1ac7c6589a37174b609e2d675717a"}
Sep 11 18:51:20.625933 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:20.625591    2582 scope.go:117] "RemoveContainer" containerID="02753ee0e3e1b1b5ebc3bda3e2c021898d171f3a72e3229b450b4531ae08eec2"
Sep 11 18:51:20.644097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:20.644072    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:51:20.650347 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:20.650325    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-461acb-openshift-default-cd7cbd856-6tsfr"]
Sep 11 18:51:22.553326 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:22.553294    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" path="/var/lib/kubelet/pods/6deb4fc4-81fd-47db-a7e5-a8ff1260968e/volumes"
Sep 11 18:51:25.542036 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.541951    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:51:25.542528 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.542204    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" podUID="b7114e69-1de6-4188-9b81-ead77d852c26" containerName="payload-processing" containerID="cri-o://a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624" gracePeriod=30
Sep 11 18:51:25.548919 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.548884    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:51:25.549435 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.549093    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" podUID="eb61ba98-345e-42c7-88c0-3993a2927851" containerName="payload-pre-processing" containerID="cri-o://48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01" gracePeriod=30
Sep 11 18:51:25.707576 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.707543    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"]
Sep 11 18:51:25.713750 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.713703    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-b01b33-lhkrm"]
Sep 11 18:51:25.828504 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.828485    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:51:25.831097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.831076    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:51:25.912605 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.912574    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume\") pod \"b7114e69-1de6-4188-9b81-ead77d852c26\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") "
Sep 11 18:51:25.912605 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.912613    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume\") pod \"eb61ba98-345e-42c7-88c0-3993a2927851\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") "
Sep 11 18:51:25.912892 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.912640    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s65vz\" (UniqueName: \"kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz\") pod \"b7114e69-1de6-4188-9b81-ead77d852c26\" (UID: \"b7114e69-1de6-4188-9b81-ead77d852c26\") "
Sep 11 18:51:25.912892 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.912663    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x24f\" (UniqueName: \"kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f\") pod \"eb61ba98-345e-42c7-88c0-3993a2927851\" (UID: \"eb61ba98-345e-42c7-88c0-3993a2927851\") "
Sep 11 18:51:25.913010 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.912984    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b7114e69-1de6-4188-9b81-ead77d852c26" (UID: "b7114e69-1de6-4188-9b81-ead77d852c26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:25.913071 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.913005    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb61ba98-345e-42c7-88c0-3993a2927851" (UID: "eb61ba98-345e-42c7-88c0-3993a2927851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:25.914743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.914711    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz" (OuterVolumeSpecName: "kube-api-access-s65vz") pod "b7114e69-1de6-4188-9b81-ead77d852c26" (UID: "b7114e69-1de6-4188-9b81-ead77d852c26"). InnerVolumeSpecName "kube-api-access-s65vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:25.914865 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:25.914774    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f" (OuterVolumeSpecName: "kube-api-access-7x24f") pod "eb61ba98-345e-42c7-88c0-3993a2927851" (UID: "eb61ba98-345e-42c7-88c0-3993a2927851"). InnerVolumeSpecName "kube-api-access-7x24f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:26.014305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.014270    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7114e69-1de6-4188-9b81-ead77d852c26-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:26.014305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.014299    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb61ba98-345e-42c7-88c0-3993a2927851-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:26.014305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.014309    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s65vz\" (UniqueName: \"kubernetes.io/projected/b7114e69-1de6-4188-9b81-ead77d852c26-kube-api-access-s65vz\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:26.014537 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.014317    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x24f\" (UniqueName: \"kubernetes.io/projected/eb61ba98-345e-42c7-88c0-3993a2927851-kube-api-access-7x24f\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:26.554032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.553990    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f2be16e-e8d4-490b-9cf1-c2a73364e05a" path="/var/lib/kubelet/pods/1f2be16e-e8d4-490b-9cf1-c2a73364e05a/volumes"
Sep 11 18:51:26.654903 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.654864    2582 generic.go:356] "Generic (PLEG): container finished" podID="eb61ba98-345e-42c7-88c0-3993a2927851" containerID="48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01" exitCode=0
Sep 11 18:51:26.655081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.654902    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" event={"ID":"eb61ba98-345e-42c7-88c0-3993a2927851","Type":"ContainerDied","Data":"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"}
Sep 11 18:51:26.655081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.654957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn" event={"ID":"eb61ba98-345e-42c7-88c0-3993a2927851","Type":"ContainerDied","Data":"0569ceea9339d2920ee62d943bb4ec68755ab0043a051b9ab9952eb98181e3fe"}
Sep 11 18:51:26.655081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.654910    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"
Sep 11 18:51:26.655081 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.654985    2582 scope.go:117] "RemoveContainer" containerID="48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"
Sep 11 18:51:26.656421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.656307    2582 generic.go:356] "Generic (PLEG): container finished" podID="b7114e69-1de6-4188-9b81-ead77d852c26" containerID="a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624" exitCode=0
Sep 11 18:51:26.656421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.656338    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" event={"ID":"b7114e69-1de6-4188-9b81-ead77d852c26","Type":"ContainerDied","Data":"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"}
Sep 11 18:51:26.656421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.656358    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"
Sep 11 18:51:26.656421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.656395    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc" event={"ID":"b7114e69-1de6-4188-9b81-ead77d852c26","Type":"ContainerDied","Data":"94132ccd52d087d1288ce0270ac7459b08f48b415cb88608e1e25604d458fc89"}
Sep 11 18:51:26.663252 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.663235    2582 scope.go:117] "RemoveContainer" containerID="48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"
Sep 11 18:51:26.663526 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:51:26.663505    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01\": container with ID starting with 48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01 not found: ID does not exist" containerID="48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"
Sep 11 18:51:26.663575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.663534    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01"} err="failed to get container status \"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01\": rpc error: code = NotFound desc = could not find container \"48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01\": container with ID starting with 48e92700cc8b2b3e39804d7074478f00048856765b38ac6c2c849d59afa4fd01 not found: ID does not exist"
Sep 11 18:51:26.663575 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.663552    2582 scope.go:117] "RemoveContainer" containerID="a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"
Sep 11 18:51:26.670680 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.670664    2582 scope.go:117] "RemoveContainer" containerID="a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"
Sep 11 18:51:26.670938 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:51:26.670918    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624\": container with ID starting with a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624 not found: ID does not exist" containerID="a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"
Sep 11 18:51:26.671026 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.670942    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624"} err="failed to get container status \"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624\": rpc error: code = NotFound desc = could not find container \"a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624\": container with ID starting with a14da930b0efe0f6cdebbe06a624c07443381e471a68e57e37da4caea0a2a624 not found: ID does not exist"
Sep 11 18:51:26.675440 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.675419    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:51:26.678921 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.678901    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b01b33-c775dff49-crkrc"]
Sep 11 18:51:26.688517 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.688490    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:51:26.692593 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:26.692574    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b01b33-6cd94f5674-rhhsn"]
Sep 11 18:51:28.553419 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:28.553389    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7114e69-1de6-4188-9b81-ead77d852c26" path="/var/lib/kubelet/pods/b7114e69-1de6-4188-9b81-ead77d852c26/volumes"
Sep 11 18:51:28.553801 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:28.553721    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb61ba98-345e-42c7-88c0-3993a2927851" path="/var/lib/kubelet/pods/eb61ba98-345e-42c7-88c0-3993a2927851/volumes"
Sep 11 18:51:32.351127 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351086    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"]
Sep 11 18:51:32.351607 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351574    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:51:32.351607 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351593    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:51:32.351607 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351603    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb61ba98-345e-42c7-88c0-3993a2927851" containerName="payload-pre-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351611    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb61ba98-345e-42c7-88c0-3993a2927851" containerName="payload-pre-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351622    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7114e69-1de6-4188-9b81-ead77d852c26" containerName="payload-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351631    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7114e69-1de6-4188-9b81-ead77d852c26" containerName="payload-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351650    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2be16e-e8d4-490b-9cf1-c2a73364e05a" containerName="revoke-keys"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351658    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2be16e-e8d4-490b-9cf1-c2a73364e05a" containerName="revoke-keys"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351736    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7114e69-1de6-4188-9b81-ead77d852c26" containerName="payload-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351748    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2be16e-e8d4-490b-9cf1-c2a73364e05a" containerName="revoke-keys"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351759    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb61ba98-345e-42c7-88c0-3993a2927851" containerName="payload-pre-processing"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351771    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:51:32.351779 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.351782    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:51:32.356622 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.356600    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:32.358764 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.358744    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hkr85\""
Sep 11 18:51:32.362988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.362965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"]
Sep 11 18:51:32.468824 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.468796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln\" (UID: \"60636241-39ca-4378-ac4f-1cfcb401c447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:32.569205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.569156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln\" (UID: \"60636241-39ca-4378-ac4f-1cfcb401c447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:32.569743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.569725    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln\" (UID: \"60636241-39ca-4378-ac4f-1cfcb401c447\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:32.667744 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.667717    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:32.943018 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:32.942994    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"]
Sep 11 18:51:32.945884 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:51:32.945852    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60636241_39ca_4378_ac4f_1cfcb401c447.slice/crio-e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da WatchSource:0}: Error finding container e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da: Status 404 returned error can't find the container with id e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da
Sep 11 18:51:33.693197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:33.693150    2582 generic.go:356] "Generic (PLEG): container finished" podID="60636241-39ca-4378-ac4f-1cfcb401c447" containerID="76fd19568e5f5be129712420e8ad522ba5a366398a922b992e320c8fc1da67fa" exitCode=0
Sep 11 18:51:33.693590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:33.693232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln" event={"ID":"60636241-39ca-4378-ac4f-1cfcb401c447","Type":"ContainerDied","Data":"76fd19568e5f5be129712420e8ad522ba5a366398a922b992e320c8fc1da67fa"}
Sep 11 18:51:33.693590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:33.693284    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln" event={"ID":"60636241-39ca-4378-ac4f-1cfcb401c447","Type":"ContainerStarted","Data":"e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da"}
Sep 11 18:51:34.822117 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:34.822091    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:34.890878 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:34.890845    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca\") pod \"60636241-39ca-4378-ac4f-1cfcb401c447\" (UID: \"60636241-39ca-4378-ac4f-1cfcb401c447\") "
Sep 11 18:51:34.891205 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:34.891159    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60636241-39ca-4378-ac4f-1cfcb401c447" (UID: "60636241-39ca-4378-ac4f-1cfcb401c447"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:34.992321 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:34.992218    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60636241-39ca-4378-ac4f-1cfcb401c447-maas-api-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:35.703112 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:35.703079    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln" event={"ID":"60636241-39ca-4378-ac4f-1cfcb401c447","Type":"ContainerDied","Data":"e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da"}
Sep 11 18:51:35.703301 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:35.703127    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e89be8e01fb80d6400600002733e637d36a8019585a5a32a85b6d2bc8cadc4da"
Sep 11 18:51:35.703301 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:35.703146    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"
Sep 11 18:51:42.960802 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:42.960766    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:51:42.961373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:42.961032    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" podUID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" containerName="payload-processing" containerID="cri-o://516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499" gracePeriod=30
Sep 11 18:51:42.976365 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:42.976339    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:51:42.976573 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:42.976529    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" podUID="459b04ed-a9d6-4390-9155-dd7672f29fc4" containerName="payload-pre-processing" containerID="cri-o://c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa" gracePeriod=30
Sep 11 18:51:43.159303 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.159269    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"]
Sep 11 18:51:43.162222 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.162196    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-ec9107-x5jln"]
Sep 11 18:51:43.238384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.238361    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:51:43.240827 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.240808    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:51:43.364875 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.364843    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume\") pod \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") "
Sep 11 18:51:43.364875 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.364879    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsm7x\" (UniqueName: \"kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x\") pod \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\" (UID: \"8cafc7f3-e44b-4c26-a9dc-34faa9302317\") "
Sep 11 18:51:43.365135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.365029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume\") pod \"459b04ed-a9d6-4390-9155-dd7672f29fc4\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") "
Sep 11 18:51:43.365135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.365102    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbcwv\" (UniqueName: \"kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv\") pod \"459b04ed-a9d6-4390-9155-dd7672f29fc4\" (UID: \"459b04ed-a9d6-4390-9155-dd7672f29fc4\") "
Sep 11 18:51:43.365337 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.365317    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cafc7f3-e44b-4c26-a9dc-34faa9302317" (UID: "8cafc7f3-e44b-4c26-a9dc-34faa9302317"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:43.365425 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.365401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "459b04ed-a9d6-4390-9155-dd7672f29fc4" (UID: "459b04ed-a9d6-4390-9155-dd7672f29fc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:51:43.366958 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.366937    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x" (OuterVolumeSpecName: "kube-api-access-nsm7x") pod "8cafc7f3-e44b-4c26-a9dc-34faa9302317" (UID: "8cafc7f3-e44b-4c26-a9dc-34faa9302317"). InnerVolumeSpecName "kube-api-access-nsm7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:43.367044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.367025    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv" (OuterVolumeSpecName: "kube-api-access-rbcwv") pod "459b04ed-a9d6-4390-9155-dd7672f29fc4" (UID: "459b04ed-a9d6-4390-9155-dd7672f29fc4"). InnerVolumeSpecName "kube-api-access-rbcwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:51:43.465988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.465933    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cafc7f3-e44b-4c26-a9dc-34faa9302317-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.465988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.465981    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsm7x\" (UniqueName: \"kubernetes.io/projected/8cafc7f3-e44b-4c26-a9dc-34faa9302317-kube-api-access-nsm7x\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.465988 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.465995    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/459b04ed-a9d6-4390-9155-dd7672f29fc4-plugins-config-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.466273 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.466006    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbcwv\" (UniqueName: \"kubernetes.io/projected/459b04ed-a9d6-4390-9155-dd7672f29fc4-kube-api-access-rbcwv\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:51:43.742340 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.742306    2582 generic.go:356] "Generic (PLEG): container finished" podID="459b04ed-a9d6-4390-9155-dd7672f29fc4" containerID="c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa" exitCode=0
Sep 11 18:51:43.742522 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.742354    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"
Sep 11 18:51:43.742522 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.742397    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" event={"ID":"459b04ed-a9d6-4390-9155-dd7672f29fc4","Type":"ContainerDied","Data":"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"}
Sep 11 18:51:43.742522 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.742452    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj" event={"ID":"459b04ed-a9d6-4390-9155-dd7672f29fc4","Type":"ContainerDied","Data":"8567cd49b884b59b8fe975c62db38398e527c66c3eb4d384ce8f2197bfce6962"}
Sep 11 18:51:43.742522 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.742479    2582 scope.go:117] "RemoveContainer" containerID="c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"
Sep 11 18:51:43.743714 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.743684    2582 generic.go:356] "Generic (PLEG): container finished" podID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" containerID="516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499" exitCode=0
Sep 11 18:51:43.743816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.743715    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" event={"ID":"8cafc7f3-e44b-4c26-a9dc-34faa9302317","Type":"ContainerDied","Data":"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"}
Sep 11 18:51:43.743816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.743745    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"
Sep 11 18:51:43.743816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.743763    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7" event={"ID":"8cafc7f3-e44b-4c26-a9dc-34faa9302317","Type":"ContainerDied","Data":"ceef95206652d23987197ee2496a2b8dde2c08269b29d33e172b194a0accad7c"}
Sep 11 18:51:43.751534 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.751507    2582 scope.go:117] "RemoveContainer" containerID="c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"
Sep 11 18:51:43.751855 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:51:43.751804    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa\": container with ID starting with c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa not found: ID does not exist" containerID="c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"
Sep 11 18:51:43.751855 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.751838    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa"} err="failed to get container status \"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa\": rpc error: code = NotFound desc = could not find container \"c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa\": container with ID starting with c5256e40a0554b446bb33f9b6bb2ada0b2505527f13b347fead06d5c642604aa not found: ID does not exist"
Sep 11 18:51:43.751971 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.751860    2582 scope.go:117] "RemoveContainer" containerID="516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"
Sep 11 18:51:43.759572 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.759555    2582 scope.go:117] "RemoveContainer" containerID="516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"
Sep 11 18:51:43.759841 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:51:43.759819    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499\": container with ID starting with 516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499 not found: ID does not exist" containerID="516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"
Sep 11 18:51:43.759929 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.759846    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499"} err="failed to get container status \"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499\": rpc error: code = NotFound desc = could not find container \"516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499\": container with ID starting with 516b5bd7ad94a1e37db7c33b789ef920c9985b149955206128c8474351d09499 not found: ID does not exist"
Sep 11 18:51:43.766809 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.766787    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:51:43.772772 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.772750    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-ec9107-688cfd7599-9twj7"]
Sep 11 18:51:43.782443 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.782425    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:51:43.787006 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:43.786985    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-ec9107-65684fd498-smqzj"]
Sep 11 18:51:44.553354 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:44.553322    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459b04ed-a9d6-4390-9155-dd7672f29fc4" path="/var/lib/kubelet/pods/459b04ed-a9d6-4390-9155-dd7672f29fc4/volumes"
Sep 11 18:51:44.553713 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:44.553640    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60636241-39ca-4378-ac4f-1cfcb401c447" path="/var/lib/kubelet/pods/60636241-39ca-4378-ac4f-1cfcb401c447/volumes"
Sep 11 18:51:44.553939 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:51:44.553927    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" path="/var/lib/kubelet/pods/8cafc7f3-e44b-4c26-a9dc-34faa9302317/volumes"
Sep 11 18:52:01.004097 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.004065    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:52:01.004683 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.004348    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" podUID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" containerName="maas-api" containerID="cri-o://9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f" gracePeriod=30
Sep 11 18:52:01.248740 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.248711    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:52:01.418880 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.418843    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gqnf\" (UniqueName: \"kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf\") pod \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") "
Sep 11 18:52:01.419080 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.418902    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs\") pod \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") "
Sep 11 18:52:01.419080 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.418917    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls\") pod \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") "
Sep 11 18:52:01.419080 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.418932    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca\") pod \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\" (UID: \"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc\") "
Sep 11 18:52:01.419323 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.419297    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" (UID: "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:52:01.421099 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.421064    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" (UID: "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:52:01.421235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.421117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf" (OuterVolumeSpecName: "kube-api-access-7gqnf") pod "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" (UID: "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc"). InnerVolumeSpecName "kube-api-access-7gqnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:52:01.421394 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.421372    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" (UID: "bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:52:01.520192 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.520138    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gqnf\" (UniqueName: \"kubernetes.io/projected/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-kube-api-access-7gqnf\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.520192 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.520188    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.520192 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.520199    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.520429 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.520207    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:52:01.828308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.828223    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" containerID="9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f" exitCode=0
Sep 11 18:52:01.828308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.828272    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"
Sep 11 18:52:01.828488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.828312    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" event={"ID":"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc","Type":"ContainerDied","Data":"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"}
Sep 11 18:52:01.828488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.828369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t" event={"ID":"bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc","Type":"ContainerDied","Data":"7468ca50f7473814f59f9a8505140be404105803f9777c775b06dbbeb846d73d"}
Sep 11 18:52:01.828488 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.828396    2582 scope.go:117] "RemoveContainer" containerID="9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"
Sep 11 18:52:01.837590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.837573    2582 scope.go:117] "RemoveContainer" containerID="9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"
Sep 11 18:52:01.837883 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:52:01.837865    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f\": container with ID starting with 9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f not found: ID does not exist" containerID="9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"
Sep 11 18:52:01.837964 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.837891    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f"} err="failed to get container status \"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f\": rpc error: code = NotFound desc = could not find container \"9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f\": container with ID starting with 9e7ea90f3fbb0006b577966ae273d5cfa6ccd25827cf6a9e769922df3843745f not found: ID does not exist"
Sep 11 18:52:01.849206 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.849185    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:52:01.859375 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:01.859354    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-d6c4cc-7d46fdf8c9-6h27t"]
Sep 11 18:52:02.559287 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:52:02.559250    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" path="/var/lib/kubelet/pods/bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc/volumes"
Sep 11 18:53:12.936568 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936530    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936799    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60636241-39ca-4378-ac4f-1cfcb401c447" containerName="revoke-keys"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936810    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="60636241-39ca-4378-ac4f-1cfcb401c447" containerName="revoke-keys"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936818    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459b04ed-a9d6-4390-9155-dd7672f29fc4" containerName="payload-pre-processing"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936823    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="459b04ed-a9d6-4390-9155-dd7672f29fc4" containerName="payload-pre-processing"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936836    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" containerName="payload-processing"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936842    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" containerName="payload-processing"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936849    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" containerName="maas-api"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936854    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" containerName="maas-api"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936870    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936875    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="6deb4fc4-81fd-47db-a7e5-a8ff1260968e" containerName="istio-proxy"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936936    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb86bcea-fe3f-46fc-bf8b-8cafe4c175dc" containerName="maas-api"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936944    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="60636241-39ca-4378-ac4f-1cfcb401c447" containerName="revoke-keys"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936952    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cafc7f3-e44b-4c26-a9dc-34faa9302317" containerName="payload-processing"
Sep 11 18:53:12.936998 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.936957    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="459b04ed-a9d6-4390-9155-dd7672f29fc4" containerName="payload-pre-processing"
Sep 11 18:53:12.939752 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.939732    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:12.943258 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.943229    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:53:12.943258 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.943242    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:53:12.943437 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.943242    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jzqpz\""
Sep 11 18:53:12.943494 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.943483    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:53:12.950514 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.950492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:12.985590 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.985565    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmwh9\" (UniqueName: \"kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:12.985692 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.985600    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:12.985692 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:12.985617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.085957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.085932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.085957 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.085962    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.086197 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.086020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmwh9\" (UniqueName: \"kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.088429 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.088407    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.088540 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.088433    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.095031 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.095006    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmwh9\" (UniqueName: \"kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9\") pod \"maas-controller-55dfcbfd4c-wtpnk\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") " pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.249333 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.249258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:13.521603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.521575    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:13.523442 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:53:13.523413    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode782ea9d_3604_4f73_bb2a_c322b7af8611.slice/crio-4acf5bdec89d9067564a4f179ff8ef06207b5b556802fa3eec55b2fcb5c766df WatchSource:0}: Error finding container 4acf5bdec89d9067564a4f179ff8ef06207b5b556802fa3eec55b2fcb5c766df: Status 404 returned error can't find the container with id 4acf5bdec89d9067564a4f179ff8ef06207b5b556802fa3eec55b2fcb5c766df
Sep 11 18:53:13.524966 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:13.524944    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 18:53:14.151260 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:14.151218    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" event={"ID":"e782ea9d-3604-4f73-bb2a-c322b7af8611","Type":"ContainerStarted","Data":"4acf5bdec89d9067564a4f179ff8ef06207b5b556802fa3eec55b2fcb5c766df"}
Sep 11 18:53:16.162085 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:16.162030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" event={"ID":"e782ea9d-3604-4f73-bb2a-c322b7af8611","Type":"ContainerStarted","Data":"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"}
Sep 11 18:53:16.162493 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:16.162107    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:16.192710 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:16.192669    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" podStartSLOduration=2.358575642 podStartE2EDuration="4.192654457s" totalImagesPullingTime="1.834078815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:16.192346249 +0000 UTC m=+2266.078318740" watchObservedRunningTime="2026-09-11 18:53:16.192654457 +0000 UTC m=+2266.078626963"
Sep 11 18:53:27.169282 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:27.169235    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:34.422497 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.422462    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:34.422974 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.422675    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" podUID="e782ea9d-3604-4f73-bb2a-c322b7af8611" containerName="manager" containerID="cri-o://e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b" gracePeriod=30
Sep 11 18:53:34.664789 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.664766    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:34.743842 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.743770    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmwh9\" (UniqueName: \"kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9\") pod \"e782ea9d-3604-4f73-bb2a-c322b7af8611\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") "
Sep 11 18:53:34.743842 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.743833    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs\") pod \"e782ea9d-3604-4f73-bb2a-c322b7af8611\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") "
Sep 11 18:53:34.744010 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.743868    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert\") pod \"e782ea9d-3604-4f73-bb2a-c322b7af8611\" (UID: \"e782ea9d-3604-4f73-bb2a-c322b7af8611\") "
Sep 11 18:53:34.745920 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.745884    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e782ea9d-3604-4f73-bb2a-c322b7af8611" (UID: "e782ea9d-3604-4f73-bb2a-c322b7af8611"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:53:34.746054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.745961    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e782ea9d-3604-4f73-bb2a-c322b7af8611" (UID: "e782ea9d-3604-4f73-bb2a-c322b7af8611"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:53:34.746054 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.745968    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9" (OuterVolumeSpecName: "kube-api-access-bmwh9") pod "e782ea9d-3604-4f73-bb2a-c322b7af8611" (UID: "e782ea9d-3604-4f73-bb2a-c322b7af8611"). InnerVolumeSpecName "kube-api-access-bmwh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:53:34.845022 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.844993    2582 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-webhook-cert\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:53:34.845022 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.845015    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmwh9\" (UniqueName: \"kubernetes.io/projected/e782ea9d-3604-4f73-bb2a-c322b7af8611-kube-api-access-bmwh9\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:53:34.845022 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:34.845025    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e782ea9d-3604-4f73-bb2a-c322b7af8611-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:53:35.250182 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.250138    2582 generic.go:356] "Generic (PLEG): container finished" podID="e782ea9d-3604-4f73-bb2a-c322b7af8611" containerID="e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b" exitCode=0
Sep 11 18:53:35.250384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.250202    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" event={"ID":"e782ea9d-3604-4f73-bb2a-c322b7af8611","Type":"ContainerDied","Data":"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"}
Sep 11 18:53:35.250384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.250220    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk"
Sep 11 18:53:35.250384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.250272    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-wtpnk" event={"ID":"e782ea9d-3604-4f73-bb2a-c322b7af8611","Type":"ContainerDied","Data":"4acf5bdec89d9067564a4f179ff8ef06207b5b556802fa3eec55b2fcb5c766df"}
Sep 11 18:53:35.250384 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.250295    2582 scope.go:117] "RemoveContainer" containerID="e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"
Sep 11 18:53:35.260120 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.260097    2582 scope.go:117] "RemoveContainer" containerID="e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"
Sep 11 18:53:35.260446 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:53:35.260424    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b\": container with ID starting with e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b not found: ID does not exist" containerID="e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"
Sep 11 18:53:35.260526 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.260458    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b"} err="failed to get container status \"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b\": rpc error: code = NotFound desc = could not find container \"e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b\": container with ID starting with e252e8afdc2e138227b28fd569f7ec1d52964f4df5147d39ea6793894a6a8a5b not found: ID does not exist"
Sep 11 18:53:35.272308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.272282    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:35.275939 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:35.275918    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-wtpnk"]
Sep 11 18:53:36.089297 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.089257    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-9p5wg"]
Sep 11 18:53:36.089677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.089537    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e782ea9d-3604-4f73-bb2a-c322b7af8611" containerName="manager"
Sep 11 18:53:36.089677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.089547    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e782ea9d-3604-4f73-bb2a-c322b7af8611" containerName="manager"
Sep 11 18:53:36.089677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.089625    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e782ea9d-3604-4f73-bb2a-c322b7af8611" containerName="manager"
Sep 11 18:53:36.093792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.093773    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.097295 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.097267    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 18:53:36.097568 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.097550    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 18:53:36.097661 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.097562    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 18:53:36.097661 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.097555    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-jzqpz\""
Sep 11 18:53:36.106633 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.106608    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-9p5wg"]
Sep 11 18:53:36.156354 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.156315    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-metrics-certs\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.156548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.156447    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnwl\" (UniqueName: \"kubernetes.io/projected/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-kube-api-access-jcnwl\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.156548 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.156498    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-webhook-cert\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.257808 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.257773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-metrics-certs\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.257991 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.257867    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnwl\" (UniqueName: \"kubernetes.io/projected/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-kube-api-access-jcnwl\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.257991 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.257909    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-webhook-cert\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.260486 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.260462    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-webhook-cert\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.260588 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.260494    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-metrics-certs\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.266579 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.266550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnwl\" (UniqueName: \"kubernetes.io/projected/c823d88a-0d6d-4e7a-a65d-4f14346d8e92-kube-api-access-jcnwl\") pod \"maas-controller-55dfcbfd4c-9p5wg\" (UID: \"c823d88a-0d6d-4e7a-a65d-4f14346d8e92\") " pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.403823 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.403789    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:36.552951 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.552914    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e782ea9d-3604-4f73-bb2a-c322b7af8611" path="/var/lib/kubelet/pods/e782ea9d-3604-4f73-bb2a-c322b7af8611/volumes"
Sep 11 18:53:36.673777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:36.673702    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-55dfcbfd4c-9p5wg"]
Sep 11 18:53:36.676583 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:53:36.676557    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc823d88a_0d6d_4e7a_a65d_4f14346d8e92.slice/crio-e7605c2de3e82a9c94589fd0b4bd864799d03b944d9d7f745d3b229e94fa2f34 WatchSource:0}: Error finding container e7605c2de3e82a9c94589fd0b4bd864799d03b944d9d7f745d3b229e94fa2f34: Status 404 returned error can't find the container with id e7605c2de3e82a9c94589fd0b4bd864799d03b944d9d7f745d3b229e94fa2f34
Sep 11 18:53:37.263596 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:37.263559    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg" event={"ID":"c823d88a-0d6d-4e7a-a65d-4f14346d8e92","Type":"ContainerStarted","Data":"7f66c0b86bf5328c59d4d6c1605c750fac2d289a6c763df6338099a46bad3491"}
Sep 11 18:53:37.263936 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:37.263639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg" event={"ID":"c823d88a-0d6d-4e7a-a65d-4f14346d8e92","Type":"ContainerStarted","Data":"e7605c2de3e82a9c94589fd0b4bd864799d03b944d9d7f745d3b229e94fa2f34"}
Sep 11 18:53:37.263936 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:37.263678    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:53:37.285247 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:37.285133    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg" podStartSLOduration=0.927521214 podStartE2EDuration="1.285116546s" totalImagesPullingTime="357.595332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:53:37.284783497 +0000 UTC m=+2287.170756011" watchObservedRunningTime="2026-09-11 18:53:37.285116546 +0000 UTC m=+2287.171089036"
Sep 11 18:53:48.269746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:53:48.269712    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-55dfcbfd4c-9p5wg"
Sep 11 18:54:41.251417 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:41.251383    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:54:41.251927 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:41.251579    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager" containerID="cri-o://cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4" gracePeriod=10
Sep 11 18:54:43.303446 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.303425    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:54:43.409374 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.409339    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume\") pod \"da90cf07-4257-47ea-807b-47629aaec120\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") "
Sep 11 18:54:43.409556 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.409405    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7z2t\" (UniqueName: \"kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t\") pod \"da90cf07-4257-47ea-807b-47629aaec120\" (UID: \"da90cf07-4257-47ea-807b-47629aaec120\") "
Sep 11 18:54:43.409735 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.409710    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "da90cf07-4257-47ea-807b-47629aaec120" (UID: "da90cf07-4257-47ea-807b-47629aaec120"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:54:43.411338 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.411315    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t" (OuterVolumeSpecName: "kube-api-access-m7z2t") pod "da90cf07-4257-47ea-807b-47629aaec120" (UID: "da90cf07-4257-47ea-807b-47629aaec120"). InnerVolumeSpecName "kube-api-access-m7z2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:54:43.510893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.510858    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/da90cf07-4257-47ea-807b-47629aaec120-extensions-socket-volume\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:54:43.510893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.510887    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7z2t\" (UniqueName: \"kubernetes.io/projected/da90cf07-4257-47ea-807b-47629aaec120-kube-api-access-m7z2t\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:54:43.558893 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.558862    2582 generic.go:356] "Generic (PLEG): container finished" podID="da90cf07-4257-47ea-807b-47629aaec120" containerID="cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4" exitCode=0
Sep 11 18:54:43.559041 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.558905    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"
Sep 11 18:54:43.559041 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.558938    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerDied","Data":"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"}
Sep 11 18:54:43.559041 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.558990    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf" event={"ID":"da90cf07-4257-47ea-807b-47629aaec120","Type":"ContainerDied","Data":"48b82300b90dc9678cf7c3a8cbc1b8adcb47f69c0757d90d79fb6b7b7571bfbf"}
Sep 11 18:54:43.559041 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.559011    2582 scope.go:117] "RemoveContainer" containerID="cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"
Sep 11 18:54:43.567722 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.567705    2582 scope.go:117] "RemoveContainer" containerID="2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"
Sep 11 18:54:43.574781 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.574765    2582 scope.go:117] "RemoveContainer" containerID="cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"
Sep 11 18:54:43.575024 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:54:43.575007    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4\": container with ID starting with cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4 not found: ID does not exist" containerID="cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"
Sep 11 18:54:43.575069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.575031    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4"} err="failed to get container status \"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4\": rpc error: code = NotFound desc = could not find container \"cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4\": container with ID starting with cd6efda531d460469470efce1f5acd1df854291d572c1b5bdb69e3af1fd67bf4 not found: ID does not exist"
Sep 11 18:54:43.575069 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.575046    2582 scope.go:117] "RemoveContainer" containerID="2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"
Sep 11 18:54:43.575318 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:54:43.575302    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1\": container with ID starting with 2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1 not found: ID does not exist" containerID="2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"
Sep 11 18:54:43.575376 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.575322    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1"} err="failed to get container status \"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1\": rpc error: code = NotFound desc = could not find container \"2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1\": container with ID starting with 2699547b8289cd2aa4f6e5fa5402b3d29936aa4f65039b69854a704e0e59e5e1 not found: ID does not exist"
Sep 11 18:54:43.580547 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.580527    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:54:43.585558 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:43.585540    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-xr5wf"]
Sep 11 18:54:44.553023 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:54:44.552989    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da90cf07-4257-47ea-807b-47629aaec120" path="/var/lib/kubelet/pods/da90cf07-4257-47ea-807b-47629aaec120/volumes"
Sep 11 18:55:41.921796 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.921763    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"]
Sep 11 18:55:41.922309 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.922039    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:55:41.922309 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.922050    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:55:41.922309 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.922108    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:55:41.922309 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.922116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:55:41.924994 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.924975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:41.927533 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.927511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xz6wx\""
Sep 11 18:55:41.939416 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.939392    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"]
Sep 11 18:55:41.993883 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.993849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6db662-06d8-48c5-8675-8fcdc918c61f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:41.994044 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:41.993908    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttnjr\" (UniqueName: \"kubernetes.io/projected/ef6db662-06d8-48c5-8675-8fcdc918c61f-kube-api-access-ttnjr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.095311 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.095275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6db662-06d8-48c5-8675-8fcdc918c61f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.095453 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.095336    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttnjr\" (UniqueName: \"kubernetes.io/projected/ef6db662-06d8-48c5-8675-8fcdc918c61f-kube-api-access-ttnjr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.095680 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.095657    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ef6db662-06d8-48c5-8675-8fcdc918c61f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.109908 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.109882    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttnjr\" (UniqueName: \"kubernetes.io/projected/ef6db662-06d8-48c5-8675-8fcdc918c61f-kube-api-access-ttnjr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7xcw8\" (UID: \"ef6db662-06d8-48c5-8675-8fcdc918c61f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.234477 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.234396    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.513906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.513799    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"]
Sep 11 18:55:42.516885 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:55:42.516856    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6db662_06d8_48c5_8675_8fcdc918c61f.slice/crio-e141bf4bdcbe6fa9908e0335745b2be3f425bffce9155767e822fe69eaa1a27c WatchSource:0}: Error finding container e141bf4bdcbe6fa9908e0335745b2be3f425bffce9155767e822fe69eaa1a27c: Status 404 returned error can't find the container with id e141bf4bdcbe6fa9908e0335745b2be3f425bffce9155767e822fe69eaa1a27c
Sep 11 18:55:42.812513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.812440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8" event={"ID":"ef6db662-06d8-48c5-8675-8fcdc918c61f","Type":"ContainerStarted","Data":"db6908c82c1bd31076c70eb7e32754643616c6e306c028642461e6046c768d1d"}
Sep 11 18:55:42.812513 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.812488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8" event={"ID":"ef6db662-06d8-48c5-8675-8fcdc918c61f","Type":"ContainerStarted","Data":"e141bf4bdcbe6fa9908e0335745b2be3f425bffce9155767e822fe69eaa1a27c"}
Sep 11 18:55:42.812705 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.812567    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:55:42.838319 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:42.838274    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8" podStartSLOduration=1.838258786 podStartE2EDuration="1.838258786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:55:42.836269557 +0000 UTC m=+2412.722242046" watchObservedRunningTime="2026-09-11 18:55:42.838258786 +0000 UTC m=+2412.724231276"
Sep 11 18:55:53.819610 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:55:53.819516    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7xcw8"
Sep 11 18:56:23.654452 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.654421    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:56:23.654912 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.654699    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:56:23.654912 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.654709    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="da90cf07-4257-47ea-807b-47629aaec120" containerName="manager"
Sep 11 18:56:23.657673 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.657656    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.659950 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.659922    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-22c08770-kserve-self-signed-certs\""
Sep 11 18:56:23.666906 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.666883    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:56:23.727519 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727490    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.727651 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.727651 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727605    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.727730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.727730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727678    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.727730 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.727697    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf69p\" (UniqueName: \"kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828471 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828515    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf69p\" (UniqueName: \"kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828658 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.828960 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828930    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.829110 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.828943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.829252 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.829229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.831191 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.831141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.831648 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.831627    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.840809 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.840782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf69p\" (UniqueName: \"kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p\") pod \"e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") " pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:23.966790 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:23.966712    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:24.240527 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:24.240443    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:56:24.244371 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:56:24.244346    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f69d211_4f66_4273_9360_d98348ba553e.slice/crio-c8e9fa69673c66d4bdb525d0d93e2f9146154fe902a2e1ac98dcb6fd60c18137 WatchSource:0}: Error finding container c8e9fa69673c66d4bdb525d0d93e2f9146154fe902a2e1ac98dcb6fd60c18137: Status 404 returned error can't find the container with id c8e9fa69673c66d4bdb525d0d93e2f9146154fe902a2e1ac98dcb6fd60c18137
Sep 11 18:56:25.003226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:25.003166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" event={"ID":"2f69d211-4f66-4273-9360-d98348ba553e","Type":"ContainerStarted","Data":"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"}
Sep 11 18:56:25.003226 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:25.003230    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" event={"ID":"2f69d211-4f66-4273-9360-d98348ba553e","Type":"ContainerStarted","Data":"c8e9fa69673c66d4bdb525d0d93e2f9146154fe902a2e1ac98dcb6fd60c18137"}
Sep 11 18:56:25.022945 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:25.022877    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" podStartSLOduration=2.022859997 podStartE2EDuration="2.022859997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:56:25.021675803 +0000 UTC m=+2454.907648293" watchObservedRunningTime="2026-09-11 18:56:25.022859997 +0000 UTC m=+2454.908832487"
Sep 11 18:56:33.967234 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:33.967193    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:33.967727 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:33.967250    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:33.979848 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:33.979821    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:56:34.055491 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:56:34.055459    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:57:10.583308 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:10.583275    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:57:25.602458 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.602353    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" podUID="2f69d211-4f66-4273-9360-d98348ba553e" containerName="main" containerID="cri-o://050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c" gracePeriod=45
Sep 11 18:57:25.857792 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.857720    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:57:25.926475 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926436    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926503    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926551    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf69p\" (UniqueName: \"kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926579    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926610    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926666 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926659    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache\") pod \"2f69d211-4f66-4273-9360-d98348ba553e\" (UID: \"2f69d211-4f66-4273-9360-d98348ba553e\") "
Sep 11 18:57:25.926961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926752    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:25.926961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926795    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home" (OuterVolumeSpecName: "home") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:25.926961 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926913    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-tmp-dir\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:25.927113 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926973    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-home\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:25.927113 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.926982    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:25.928935 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.928911    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:57:25.929106 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.929087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm" (OuterVolumeSpecName: "dshm") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 18:57:25.929150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:25.929100    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p" (OuterVolumeSpecName: "kube-api-access-jf69p") pod "2f69d211-4f66-4273-9360-d98348ba553e" (UID: "2f69d211-4f66-4273-9360-d98348ba553e"). InnerVolumeSpecName "kube-api-access-jf69p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:57:26.028245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.028199    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf69p\" (UniqueName: \"kubernetes.io/projected/2f69d211-4f66-4273-9360-d98348ba553e-kube-api-access-jf69p\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:26.028245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.028235    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-dshm\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:26.028245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.028245    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f69d211-4f66-4273-9360-d98348ba553e-tls-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:26.028245 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.028254    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f69d211-4f66-4273-9360-d98348ba553e-model-cache\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:26.273292 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.273253    2582 generic.go:356] "Generic (PLEG): container finished" podID="2f69d211-4f66-4273-9360-d98348ba553e" containerID="050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c" exitCode=0
Sep 11 18:57:26.273490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.273301    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"
Sep 11 18:57:26.273490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.273343    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" event={"ID":"2f69d211-4f66-4273-9360-d98348ba553e","Type":"ContainerDied","Data":"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"}
Sep 11 18:57:26.273490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.273418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7" event={"ID":"2f69d211-4f66-4273-9360-d98348ba553e","Type":"ContainerDied","Data":"c8e9fa69673c66d4bdb525d0d93e2f9146154fe902a2e1ac98dcb6fd60c18137"}
Sep 11 18:57:26.273490 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.273440    2582 scope.go:117] "RemoveContainer" containerID="050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"
Sep 11 18:57:26.282029 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.282013    2582 scope.go:117] "RemoveContainer" containerID="050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"
Sep 11 18:57:26.282313 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:57:26.282293    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c\": container with ID starting with 050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c not found: ID does not exist" containerID="050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"
Sep 11 18:57:26.282365 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.282321    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c"} err="failed to get container status \"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c\": rpc error: code = NotFound desc = could not find container \"050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c\": container with ID starting with 050407735d3cf5e01cbbf9d404e98bca72250d108138314287dd1ed722eb8f0c not found: ID does not exist"
Sep 11 18:57:26.295566 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.295542    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:57:26.301466 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.301443    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-22c08770-kserve-c75fb845f-2drk7"]
Sep 11 18:57:26.552652 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:26.552562    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f69d211-4f66-4273-9360-d98348ba553e" path="/var/lib/kubelet/pods/2f69d211-4f66-4273-9360-d98348ba553e/volumes"
Sep 11 18:57:30.999940 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:30.999911    2582 scope.go:117] "RemoveContainer" containerID="3b2f8e13d50a659b7faf32778a0d559a2920d0047d14dceb1bbfba08eb8990f8"
Sep 11 18:57:32.244572 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.244534    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:32.245019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.244960    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f69d211-4f66-4273-9360-d98348ba553e" containerName="main"
Sep 11 18:57:32.245019 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.244977    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f69d211-4f66-4273-9360-d98348ba553e" containerName="main"
Sep 11 18:57:32.245141 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.245057    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f69d211-4f66-4273-9360-d98348ba553e" containerName="main"
Sep 11 18:57:32.249416 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.249393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.252770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.252736    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 11 18:57:32.252770 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.252747    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lvtx6\""
Sep 11 18:57:32.252930 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.252736    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 11 18:57:32.252930 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.252794    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 11 18:57:32.252930 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.252813    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 18:57:32.257459 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.257436    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:32.266517 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.266494    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"]
Sep 11 18:57:32.269547 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.269529    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.284258 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.284226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"]
Sep 11 18:57:32.378391 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s42mj\" (UniqueName: \"kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.378583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378410    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.378583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.378583 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378513    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.378757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378582    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-maas-api-tls\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.378757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvx8v\" (UniqueName: \"kubernetes.io/projected/016bf1da-b790-4c9e-933e-a57d6022ad08-kube-api-access-cvx8v\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.378757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016bf1da-b790-4c9e-933e-a57d6022ad08-openshift-service-ca\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.378757 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.378684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-metrics-certs\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.480056 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480028    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.480228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.480228 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480091    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.480373 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480351    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-maas-api-tls\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.480430 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvx8v\" (UniqueName: \"kubernetes.io/projected/016bf1da-b790-4c9e-933e-a57d6022ad08-kube-api-access-cvx8v\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.480489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016bf1da-b790-4c9e-933e-a57d6022ad08-openshift-service-ca\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.480489 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-metrics-certs\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.480595 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480496    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s42mj\" (UniqueName: \"kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.480841 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.480817    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.481338 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.481311    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/016bf1da-b790-4c9e-933e-a57d6022ad08-openshift-service-ca\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.482777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.482754    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-maas-api-tls\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.482777 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.482769    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.483037 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.483016    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.483084 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.483051    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016bf1da-b790-4c9e-933e-a57d6022ad08-metrics-certs\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.489987 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.489962    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvx8v\" (UniqueName: \"kubernetes.io/projected/016bf1da-b790-4c9e-933e-a57d6022ad08-kube-api-access-cvx8v\") pod \"maas-api-8559f566dc-qphlf\" (UID: \"016bf1da-b790-4c9e-933e-a57d6022ad08\") " pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.490108 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.490092    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s42mj\" (UniqueName: \"kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj\") pod \"maas-api-6598bdcc8c-d9m74\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") " pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.560681 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.560620    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:32.578612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.578588    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:32.838487 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.838463    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:32.840881 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:57:32.840851    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ca0c44_550a_409a_83e3_5c2c78a34ab1.slice/crio-546823a31cbb10a60d1933391c0a4500fa7dd7d49ec43a4a0b79815250b5ad0d WatchSource:0}: Error finding container 546823a31cbb10a60d1933391c0a4500fa7dd7d49ec43a4a0b79815250b5ad0d: Status 404 returned error can't find the container with id 546823a31cbb10a60d1933391c0a4500fa7dd7d49ec43a4a0b79815250b5ad0d
Sep 11 18:57:32.868986 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:32.868960    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"]
Sep 11 18:57:32.871394 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:57:32.871364    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016bf1da_b790_4c9e_933e_a57d6022ad08.slice/crio-8d8c325d12d6dc1caeff269cf3625946f844e268252c0890f79e70141a0d244e WatchSource:0}: Error finding container 8d8c325d12d6dc1caeff269cf3625946f844e268252c0890f79e70141a0d244e: Status 404 returned error can't find the container with id 8d8c325d12d6dc1caeff269cf3625946f844e268252c0890f79e70141a0d244e
Sep 11 18:57:33.306805 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:33.306770    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf" event={"ID":"016bf1da-b790-4c9e-933e-a57d6022ad08","Type":"ContainerStarted","Data":"8d8c325d12d6dc1caeff269cf3625946f844e268252c0890f79e70141a0d244e"}
Sep 11 18:57:33.307853 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:33.307833    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" event={"ID":"62ca0c44-550a-409a-83e3-5c2c78a34ab1","Type":"ContainerStarted","Data":"546823a31cbb10a60d1933391c0a4500fa7dd7d49ec43a4a0b79815250b5ad0d"}
Sep 11 18:57:34.313536 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.313487    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" event={"ID":"62ca0c44-550a-409a-83e3-5c2c78a34ab1","Type":"ContainerStarted","Data":"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"}
Sep 11 18:57:34.313949 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.313659    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:34.314999 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.314975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf" event={"ID":"016bf1da-b790-4c9e-933e-a57d6022ad08","Type":"ContainerStarted","Data":"d0a5856e0a41ee8ac0c1d8c95aba093b661feeaccb28eb669ebbaefc34031f0e"}
Sep 11 18:57:34.315130 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.315076    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:34.330815 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.330759    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" podStartSLOduration=1.7869381130000002 podStartE2EDuration="2.330740549s" totalImagesPullingTime="543.802436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:57:34.329351967 +0000 UTC m=+2524.215324456" watchObservedRunningTime="2026-09-11 18:57:34.330740549 +0000 UTC m=+2524.216713039"
Sep 11 18:57:34.346519 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:34.346471    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf" podStartSLOduration=1.828752067 podStartE2EDuration="2.346457923s" totalImagesPullingTime="517.705856ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 18:57:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:57:34.345311979 +0000 UTC m=+2524.231284493" watchObservedRunningTime="2026-09-11 18:57:34.346457923 +0000 UTC m=+2524.232430411"
Sep 11 18:57:40.324612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.324579    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8559f566dc-qphlf"
Sep 11 18:57:40.325123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.325043    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:40.389643 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.389615    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:40.389844 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.389820    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" podUID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" containerName="maas-api" containerID="cri-o://8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba" gracePeriod=30
Sep 11 18:57:40.633372 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.633345    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:40.750562 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.750528    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs\") pod \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") "
Sep 11 18:57:40.750723 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.750646    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls\") pod \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") "
Sep 11 18:57:40.750723 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.750685    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s42mj\" (UniqueName: \"kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj\") pod \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") "
Sep 11 18:57:40.750723 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.750719    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca\") pod \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\" (UID: \"62ca0c44-550a-409a-83e3-5c2c78a34ab1\") "
Sep 11 18:57:40.751377 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.751344    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62ca0c44-550a-409a-83e3-5c2c78a34ab1" (UID: "62ca0c44-550a-409a-83e3-5c2c78a34ab1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 18:57:40.753721 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.753691    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj" (OuterVolumeSpecName: "kube-api-access-s42mj") pod "62ca0c44-550a-409a-83e3-5c2c78a34ab1" (UID: "62ca0c44-550a-409a-83e3-5c2c78a34ab1"). InnerVolumeSpecName "kube-api-access-s42mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 18:57:40.755655 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.755626    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62ca0c44-550a-409a-83e3-5c2c78a34ab1" (UID: "62ca0c44-550a-409a-83e3-5c2c78a34ab1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:57:40.758618 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.758591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62ca0c44-550a-409a-83e3-5c2c78a34ab1" (UID: "62ca0c44-550a-409a-83e3-5c2c78a34ab1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 18:57:40.852479 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.852452    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-maas-api-tls\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:40.852592 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.852496    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s42mj\" (UniqueName: \"kubernetes.io/projected/62ca0c44-550a-409a-83e3-5c2c78a34ab1-kube-api-access-s42mj\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:40.852592 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.852506    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ca0c44-550a-409a-83e3-5c2c78a34ab1-openshift-service-ca\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:40.852592 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:40.852514    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62ca0c44-550a-409a-83e3-5c2c78a34ab1-metrics-certs\") on node \"ip-10-0-128-148.ec2.internal\" DevicePath \"\""
Sep 11 18:57:41.347753 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.347714    2582 generic.go:356] "Generic (PLEG): container finished" podID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" containerID="8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba" exitCode=0
Sep 11 18:57:41.348133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.347772    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"
Sep 11 18:57:41.348133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.347795    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" event={"ID":"62ca0c44-550a-409a-83e3-5c2c78a34ab1","Type":"ContainerDied","Data":"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"}
Sep 11 18:57:41.348133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.347866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74" event={"ID":"62ca0c44-550a-409a-83e3-5c2c78a34ab1","Type":"ContainerDied","Data":"546823a31cbb10a60d1933391c0a4500fa7dd7d49ec43a4a0b79815250b5ad0d"}
Sep 11 18:57:41.348133 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.347889    2582 scope.go:117] "RemoveContainer" containerID="8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"
Sep 11 18:57:41.356851 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.356831    2582 scope.go:117] "RemoveContainer" containerID="8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"
Sep 11 18:57:41.357083 ip-10-0-128-148 kubenswrapper[2582]: E0911 18:57:41.357059    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba\": container with ID starting with 8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba not found: ID does not exist" containerID="8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"
Sep 11 18:57:41.357130 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.357090    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba"} err="failed to get container status \"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba\": rpc error: code = NotFound desc = could not find container \"8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba\": container with ID starting with 8e7a684c8b30f787b836a2cb5a91df9656ca95c6a0ab4444ed6ac5237f02b7ba not found: ID does not exist"
Sep 11 18:57:41.369083 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.369061    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:41.375123 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:41.375104    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6598bdcc8c-d9m74"]
Sep 11 18:57:42.553271 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:57:42.553238    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" path="/var/lib/kubelet/pods/62ca0c44-550a-409a-83e3-5c2c78a34ab1/volumes"
Sep 11 18:58:07.140994 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:07.140962    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4s97g_e58588cd-c8c7-4779-b4cb-ae35dd2f4703/manager/0.log"
Sep 11 18:58:07.255936 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:07.255907    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55dfcbfd4c-9p5wg_c823d88a-0d6d-4e7a-a65d-4f14346d8e92/manager/0.log"
Sep 11 18:58:07.845051 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:07.845024    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd_e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4/copy-manifests/0.log"
Sep 11 18:58:07.851037 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:07.851018    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd_e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4/manager/0.log"
Sep 11 18:58:08.170524 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:08.170500    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8559f566dc-qphlf_016bf1da-b790-4c9e-933e-a57d6022ad08/maas-api/0.log"
Sep 11 18:58:08.496305 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:08.496228    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819205-7krzm_76b9662d-7dc0-44a1-9786-90418bf972ec/cleanup/0.log"
Sep 11 18:58:09.461005 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:09.460973    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sdbm5_bc24f7e2-c782-4fdf-961b-edfde6fbdd4c/manager/0.log"
Sep 11 18:58:09.782389 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:09.782316    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7xcw8_ef6db662-06d8-48c5-8675-8fcdc918c61f/manager/0.log"
Sep 11 18:58:10.466000 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:10.465974    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mpvbg_dbfa15c8-83d8-4812-a12b-ffb6f84f47c2/discovery/0.log"
Sep 11 18:58:10.574954 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:10.574928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-8dc6w_d02f1a7a-bec9-449e-bf7a-b9e94a5360b7/kube-auth-proxy/0.log"
Sep 11 18:58:10.912675 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:10.912646    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lcqqd_be7daac5-40f4-4aea-b77b-54ac139c013b/payload-pre-processing/0.log"
Sep 11 18:58:11.472150 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:11.472119    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-k5mh7_96e8ae04-711f-4acb-84a9-a241e774aec5/main/0.log"
Sep 11 18:58:11.706907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:11.706880    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-ffdv7_bc958ea4-57f3-4561-9d94-bc4821f8fb3a/main/0.log"
Sep 11 18:58:20.707774 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:20.707742    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kd2j8_0922c1e3-dc51-4b8c-ae1e-ac9af86c54e6/global-pull-secret-syncer/0.log"
Sep 11 18:58:20.874442 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:20.874412    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qdmqh_2aeadb79-70da-4d8e-a32c-276592debddb/konnectivity-agent/0.log"
Sep 11 18:58:20.894911 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:20.894885    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-148.ec2.internal_f1353260c7ace9243a51756424e4b1aa/haproxy/0.log"
Sep 11 18:58:26.049235 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:26.049204    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-sdbm5_bc24f7e2-c782-4fdf-961b-edfde6fbdd4c/manager/0.log"
Sep 11 18:58:26.235508 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:26.235482    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7xcw8_ef6db662-06d8-48c5-8675-8fcdc918c61f/manager/0.log"
Sep 11 18:58:28.250082 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:28.250053    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/node-exporter/0.log"
Sep 11 18:58:28.272393 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:28.272370    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/kube-rbac-proxy/0.log"
Sep 11 18:58:28.295877 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:28.295852    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7z5vk_5e63bfa1-1ee1-428d-a490-90ab6ebac60a/init-textfile/0.log"
Sep 11 18:58:30.393356 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.393322    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"]
Sep 11 18:58:30.393743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.393622    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" containerName="maas-api"
Sep 11 18:58:30.393743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.393632    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" containerName="maas-api"
Sep 11 18:58:30.393743 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.393690    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="62ca0c44-550a-409a-83e3-5c2c78a34ab1" containerName="maas-api"
Sep 11 18:58:30.396612 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.396594    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.398924 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.398903    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"openshift-service-ca.crt\""
Sep 11 18:58:30.399052 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.398974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"kube-root-ca.crt\""
Sep 11 18:58:30.399721 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.399707    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9k92\"/\"default-dockercfg-bfrpt\""
Sep 11 18:58:30.409276 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.409253    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"]
Sep 11 18:58:30.442552 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.442521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j256\" (UniqueName: \"kubernetes.io/projected/331787b8-4c8c-4017-adcc-a82c0e5c65ee-kube-api-access-6j256\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.442671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.442571    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-podres\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.442671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.442628    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-proc\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.442671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.442643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-sys\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.442671 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.442660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-lib-modules\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543564 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j256\" (UniqueName: \"kubernetes.io/projected/331787b8-4c8c-4017-adcc-a82c0e5c65ee-kube-api-access-6j256\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543717 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543578    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-podres\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543717 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543606    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-proc\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543717 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543622    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-sys\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543717 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543640    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-lib-modules\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543715    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-proc\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543723    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-sys\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-lib-modules\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.543856 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.543788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/331787b8-4c8c-4017-adcc-a82c0e5c65ee-podres\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.553907 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.553883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"kube-root-ca.crt\""
Sep 11 18:58:30.564120 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.564101    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r9k92\"/\"openshift-service-ca.crt\""
Sep 11 18:58:30.574693 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.574673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j256\" (UniqueName: \"kubernetes.io/projected/331787b8-4c8c-4017-adcc-a82c0e5c65ee-kube-api-access-6j256\") pod \"perf-node-gather-daemonset-4gxdg\" (UID: \"331787b8-4c8c-4017-adcc-a82c0e5c65ee\") " pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.708659 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.708634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r9k92\"/\"default-dockercfg-bfrpt\""
Sep 11 18:58:30.716855 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.716837    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:30.990789 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:30.990767    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"]
Sep 11 18:58:30.992352 ip-10-0-128-148 kubenswrapper[2582]: W0911 18:58:30.992312    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod331787b8_4c8c_4017_adcc_a82c0e5c65ee.slice/crio-f94275de01218b44d70888e90965c9af22c8ab7881bc76d5a213798ef73b3915 WatchSource:0}: Error finding container f94275de01218b44d70888e90965c9af22c8ab7881bc76d5a213798ef73b3915: Status 404 returned error can't find the container with id f94275de01218b44d70888e90965c9af22c8ab7881bc76d5a213798ef73b3915
Sep 11 18:58:31.019985 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:31.019962    2582 scope.go:117] "RemoveContainer" containerID="76fd19568e5f5be129712420e8ad522ba5a366398a922b992e320c8fc1da67fa"
Sep 11 18:58:31.564746 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:31.564710    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg" event={"ID":"331787b8-4c8c-4017-adcc-a82c0e5c65ee","Type":"ContainerStarted","Data":"ea1050dc99629f3ae3c5b9b848afbcf0c5cc701cd7991afe58e722c3b0bc4d89"}
Sep 11 18:58:31.565254 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:31.564759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg" event={"ID":"331787b8-4c8c-4017-adcc-a82c0e5c65ee","Type":"ContainerStarted","Data":"f94275de01218b44d70888e90965c9af22c8ab7881bc76d5a213798ef73b3915"}
Sep 11 18:58:31.565254 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:31.564792    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:31.583874 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:31.583830    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg" podStartSLOduration=1.583815797 podStartE2EDuration="1.583815797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 18:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 18:58:31.583004883 +0000 UTC m=+2581.468977371" watchObservedRunningTime="2026-09-11 18:58:31.583815797 +0000 UTC m=+2581.469788285"
Sep 11 18:58:33.283423 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:33.283393    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98h96_02b8c5e6-beb4-4aca-a918-284fd81f9210/dns/0.log"
Sep 11 18:58:33.310989 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:33.310954    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-98h96_02b8c5e6-beb4-4aca-a918-284fd81f9210/kube-rbac-proxy/0.log"
Sep 11 18:58:33.509135 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:33.509106    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s7h6h_44dd8471-2506-4cc9-9c44-9f9572550751/dns-node-resolver/0.log"
Sep 11 18:58:34.096603 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:34.096573    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-n6pjd_a948bb34-43b3-4f19-8cd0-0851b7f71882/node-ca/0.log"
Sep 11 18:58:35.474685 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:35.474657    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mpvbg_dbfa15c8-83d8-4812-a12b-ffb6f84f47c2/discovery/0.log"
Sep 11 18:58:35.495094 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:35.495075    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-8dc6w_d02f1a7a-bec9-449e-bf7a-b9e94a5360b7/kube-auth-proxy/0.log"
Sep 11 18:58:35.585975 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:35.585949    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lcqqd_be7daac5-40f4-4aea-b77b-54ac139c013b/payload-pre-processing/0.log"
Sep 11 18:58:36.574408 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:36.574374    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wlzz5_cc1f7966-16e2-47a2-98c5-084115142030/serve-healthcheck-canary/0.log"
Sep 11 18:58:37.061677 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:37.061639    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dd9n_d7bc0426-d3ab-4adc-9160-b6c1bb0fd824/kube-rbac-proxy/0.log"
Sep 11 18:58:37.083571 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:37.083538    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dd9n_d7bc0426-d3ab-4adc-9160-b6c1bb0fd824/exporter/0.log"
Sep 11 18:58:37.104518 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:37.104493    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-6dd9n_d7bc0426-d3ab-4adc-9160-b6c1bb0fd824/extractor/0.log"
Sep 11 18:58:37.578281 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:37.578251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r9k92/perf-node-gather-daemonset-4gxdg"
Sep 11 18:58:39.802262 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:39.802160    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4s97g_e58588cd-c8c7-4779-b4cb-ae35dd2f4703/manager/0.log"
Sep 11 18:58:39.849286 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:39.849253    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-55dfcbfd4c-9p5wg_c823d88a-0d6d-4e7a-a65d-4f14346d8e92/manager/0.log"
Sep 11 18:58:40.248032 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:40.248007    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd_e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4/manager/0.log"
Sep 11 18:58:40.271296 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:40.271273    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68f4cbcd7d-vfbfd_e0aa2b12-69bc-40f2-b2a8-c11bf076dbc4/copy-manifests/0.log"
Sep 11 18:58:42.035841 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:42.035810    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mpvbg_dbfa15c8-83d8-4812-a12b-ffb6f84f47c2/discovery/0.log"
Sep 11 18:58:42.047766 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:42.047734    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-8dc6w_d02f1a7a-bec9-449e-bf7a-b9e94a5360b7/kube-auth-proxy/0.log"
Sep 11 18:58:42.117024 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:42.116993    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lcqqd_be7daac5-40f4-4aea-b77b-54ac139c013b/payload-pre-processing/0.log"
Sep 11 18:58:44.033421 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:44.033388    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mpvbg_dbfa15c8-83d8-4812-a12b-ffb6f84f47c2/discovery/0.log"
Sep 11 18:58:44.057507 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:44.057484    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78f85fb5d-8dc6w_d02f1a7a-bec9-449e-bf7a-b9e94a5360b7/kube-auth-proxy/0.log"
Sep 11 18:58:44.127816 ip-10-0-128-148 kubenswrapper[2582]: I0911 18:58:44.127779    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-lcqqd_be7daac5-40f4-4aea-b77b-54ac139c013b/payload-pre-processing/0.log"