Sep 15 20:34:45.405517 ip-10-0-138-122 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 15 20:34:45.405654 ip-10-0-138-122 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 15 20:34:45.405739 ip-10-0-138-122 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 15 20:34:45.406280 ip-10-0-138-122 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 15 20:34:55.634187 ip-10-0-138-122 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 15 20:34:55.634208 ip-10-0-138-122 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e78a1a1a1b7b4f349d4fb9e608cdb06d --
Sep 15 20:36:47.264956 ip-10-0-138-122 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 20:36:47.633948 ip-10-0-138-122 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:36:47.633948 ip-10-0-138-122 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 20:36:47.633948 ip-10-0-138-122 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:36:47.633948 ip-10-0-138-122 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 20:36:47.633948 ip-10-0-138-122 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 20:36:47.636619 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.636539    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 20:36:47.640381 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640365    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.640381 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640381    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640385    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640388    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640392    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640395    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640398    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640401    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640403    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640406    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640409    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640411    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640414    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640416    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640420    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640423    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640425    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640433    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640436    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640439    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640441    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.640446 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640444    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640447    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640449    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640452    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640454    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640457    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640460    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640463    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640466    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640468    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640471    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640473    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640476    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640479    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640482    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640484    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640487    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640489    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640492    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.640949 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640494    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640498    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640502    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640505    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640508    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640511    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640514    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640517    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640520    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640522    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640527    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640531    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640534    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640537    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640540    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640544    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640546    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640549    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640551    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.641457 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640554    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640557    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640560    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640562    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640565    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640567    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640569    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640572    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640575    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640577    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640579    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640582    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640585    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640587    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640590    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640592    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640595    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640599    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640601    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640604    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.641950 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640606    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640611    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640614    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640617    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640620    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640623    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640625    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640628    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640631    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640634    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640637    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640639    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640642    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640645    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640647    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640649    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640652    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640654    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640657    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.642434 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.640659    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642111    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642119    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642123    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642127    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642130    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642133    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642136    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642139    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642141    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642144    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642146    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642149    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642153    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642156    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642158    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642161    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642163    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642166    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642168    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.642923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642171    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642173    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642176    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642178    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642180    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642183    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642185    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642188    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642190    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642194    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642198    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642201    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642203    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642206    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642208    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642211    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642213    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642215    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642218    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.643455 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642221    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642223    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642226    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642228    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642231    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642233    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642237    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642240    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642242    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642245    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642248    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642250    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642253    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642255    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642258    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642260    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642263    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642265    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642268    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642270    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.643955 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642273    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642275    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642278    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642281    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642284    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642286    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642288    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642291    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642293    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642296    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642298    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642300    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642304    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642307    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642310    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642312    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642316    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642319    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642323    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642326    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.644561 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642329    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642332    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642335    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642338    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642341    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642343    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642346    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642349    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642351    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642354    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642356    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642360    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642363    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642366    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642369    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642371    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642374    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642377    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642379    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.645071 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642382    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642384    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642761    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642766    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642769    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642772    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642775    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642778    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642780    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642783    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642785    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642789    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642793    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642797    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642801    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642805    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642807    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642810    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642813    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.645538 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642816    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642818    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642821    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642823    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642826    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642828    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642831    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642833    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642836    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642838    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642841    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642843    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642845    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642848    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642851    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642853    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642856    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642860    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642862    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.646030 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642865    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642867    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642870    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642872    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642875    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642880    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642882    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642885    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642887    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642890    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642893    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642895    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642898    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642900    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642902    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642905    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642907    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642910    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642912    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642915    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.646611 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642917    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642920    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642922    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642925    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642927    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642930    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642932    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642935    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642938    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642940    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642943    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642945    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642947    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642950    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642953    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642957    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642959    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642963    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642966    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.647123 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642968    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642971    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642976    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642979    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642981    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642984    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642986    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642989    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642991    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642994    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642996    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.642999    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643001    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643003    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643006    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643008    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643011    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643013    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643016    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.647626 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643018    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643020    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643023    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643025    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643107    2579 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643156    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643163    2579 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643169    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643175    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643178    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643183    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643188    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643191    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643194    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643197    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643200    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643206    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643209    2579 flags.go:64] FLAG: --cgroup-root=""
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643212    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643215    2579 flags.go:64] FLAG: --client-ca-file=""
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643218    2579 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643221    2579 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643225    2579 flags.go:64] FLAG: --cluster-domain=""
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643228    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 20:36:47.648128 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643231    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643234    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643237    2579 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643240    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643243    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643246    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643249    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643252    2579 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643255    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643258    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643261    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643264    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643268    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643271    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643274    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643277    2579 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643281    2579 flags.go:64] FLAG: --enable-server="true"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643285    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643290    2579 flags.go:64] FLAG: --event-burst="100"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643293    2579 flags.go:64] FLAG: --event-qps="50"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643296    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643299    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643301    2579 flags.go:64] FLAG: --eviction-hard=""
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643305    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 20:36:47.648757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643308    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643311    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643314    2579 flags.go:64] FLAG: --eviction-soft=""
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643317    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643320    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643323    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643325    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643328    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643331    2579 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643334    2579 flags.go:64] FLAG: --feature-gates=""
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643337    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643340    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643343    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643346    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643349    2579 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643352    2579 flags.go:64] FLAG: --help="false"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643355    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643358    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643360    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643363    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643367    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643370    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643372    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 20:36:47.649333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643376    2579 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643379    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643382    2579 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643387    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643390    2579 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643393    2579 flags.go:64] FLAG: --kube-reserved=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643395    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643398    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643401    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643403    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643406    2579 flags.go:64] FLAG: --lock-file=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643409    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643412    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643414    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643420    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643423    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643425    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643428    2579 flags.go:64] FLAG: --logging-format="text"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643431    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643434    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643451    2579 flags.go:64] FLAG: --manifest-url=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643454    2579 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643459    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643462    2579 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643467    2579 flags.go:64] FLAG: --max-pods="110"
Sep 15 20:36:47.649971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643470    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643473    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643475    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643478    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643481    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643484    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643487    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643511    2579 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643514    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643517    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643522    2579 flags.go:64] FLAG: --pod-cidr=""
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643525    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643530    2579 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643533    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643536    2579 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643539    2579 flags.go:64] FLAG: --port="10250"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643542    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643544    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-013bef7789d91682c"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643547    2579 flags.go:64] FLAG: --qos-reserved=""
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643550    2579 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643553    2579 flags.go:64] FLAG: --register-node="true"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643556    2579 flags.go:64] FLAG: --register-with-taints=""
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643559    2579 flags.go:64] FLAG: --registry-burst="10"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643562    2579 flags.go:64] FLAG: --registry-qps="5"
Sep 15 20:36:47.650602 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643565    2579 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643567    2579 flags.go:64] FLAG: --reserved-memory=""
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643571    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643574    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643577    2579 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643579    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643582    2579 flags.go:64] FLAG: --runonce="false"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643585    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643588    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643591    2579 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643593    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643596    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643599    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643602    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643605    2579 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643607    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643611    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643614    2579 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643618    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643621    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643625    2579 flags.go:64] FLAG: --system-cgroups=""
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643627    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643632    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643635    2579 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643638    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 20:36:47.651206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643642    2579 flags.go:64] FLAG: --tls-min-version=""
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643644    2579 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643647    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643650    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643653    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643655    2579 flags.go:64] FLAG: --v="2"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643659    2579 flags.go:64] FLAG: --version="false"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643678    2579 flags.go:64] FLAG: --vmodule=""
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643682    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.643685    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643824    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643829    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643832    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643836    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643839    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643841    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643844    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643847    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643850    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643852    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643855    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643858    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.651852 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643860    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643864    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643867    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643871    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643874    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643876    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643879    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643882    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643886    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643890    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643893    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643896    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643899    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643902    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643905    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643908    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643911    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643913    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643916    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643919    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.652382 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643921    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643924    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643927    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643929    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643932    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643935    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643937    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643940    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643942    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643945    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643948    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643950    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643952    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643956    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643959    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643963    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643966    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643969    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643971    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643974    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.652909 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643977    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643980    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643983    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643985    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643988    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643990    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643993    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643995    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.643998    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644000    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644003    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644005    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644008    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644010    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644013    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644015    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644018    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644020    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644023    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.653404 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644026    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644029    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644031    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644034    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644036    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644039    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644043    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644047    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644051    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644054    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644057    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644059    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644062    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644065    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644068    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644070    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644073    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644076    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644078    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.653957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644080    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644083    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644086    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644088    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644091    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644093    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644095    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644099    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.644102    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.644592    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.651821    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.651834    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.651853    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.651859    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 20:36:47.654423 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651915    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651920    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651923    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651927    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651930    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651932    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651935    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651938    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651941    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651943    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651946    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651949    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651952    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651955    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651957    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651960    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651964    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651968    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651971    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651973    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.654786 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651976    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651979    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651981    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651984    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651987    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651991    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651996    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.651999    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652002    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652006    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652009    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652012    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652015    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652018    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652021    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652023    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652026    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652028    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652031    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.655271 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652034    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652036    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652039    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652042    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652044    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652047    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652050    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652053    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652055    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652058    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652060    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652063    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652066    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652069    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652071    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652074    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652076    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652079    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652081    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652084    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.655755 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652087    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652089    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652092    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652096    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652098    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652102    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652104    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652107    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652110    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652113    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652115    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652119    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652121    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652124    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652127    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652129    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652132    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652134    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652137    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652140    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.656251 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652142    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652145    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652147    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652150    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652153    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652155    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652160    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652164    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652167    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652171    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652174    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652177    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652180    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652183    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652185    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652188    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652191    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652193    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652196    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.656774 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652198    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.652203    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652311    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652316    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652320    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652323    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652326    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652328    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652331    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652333    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652336    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652338    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652341    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652345    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 20:36:47.657248 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652349    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652352    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652355    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652358    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652361    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652364    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652366    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652369    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652372    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652374    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652377    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652379    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652382    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652384    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652387    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652390    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652393    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652395    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652398    2579 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652401    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 20:36:47.657609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652403    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652406    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652408    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652410    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652413    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652416    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652418    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652421    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652423    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652426    2579 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652428    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652431    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652434    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652437    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652440    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652442    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652446    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652450    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652454    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652456    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 20:36:47.658253 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652459    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652462    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652464    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652468    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652471    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652474    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652477    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652480    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652483    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652486    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652489    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652492    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652494    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652496    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652499    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652501    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652504    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652507    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652509    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 20:36:47.659158 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652512    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652515    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652517    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652520    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652522    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652525    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652527    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652530    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652533    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652535    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652538    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652541    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652543    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652546    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652549    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652551    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652554    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652556    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652559    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652561    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 20:36:47.659792 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652563    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652566    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652569    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652571    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652573    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652576    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652579    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:47.652581    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.652586    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.652699    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.654633    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.655507    2579 server.go:1019] "Starting client certificate rotation"
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.655604    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.655635    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 20:36:47.660292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.659457    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 20:36:47.664616 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.664599    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 20:36:47.674800 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.674784    2579 log.go:25] "Validated CRI v1 runtime API"
Sep 15 20:36:47.678125 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.678108    2579 log.go:25] "Validated CRI v1 image API"
Sep 15 20:36:47.679451 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.679435    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 20:36:47.682363 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.682345    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 92c8ef8b-b72d-498d-abaf-7eb0bdcbb7f6:/dev/nvme0n1p3 f262a2c3-bfd2-45b4-9265-8d16c80f13a2:/dev/nvme0n1p4]
Sep 15 20:36:47.682424 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.682363    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 20:36:47.687598 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.687579    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 20:36:47.688058 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.687951    2579 manager.go:216] Machine: {Timestamp:2026-09-15 20:36:47.686652972 +0000 UTC m=+0.313120865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098011 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c486ee678703882205acab0d57d72 SystemUUID:ec2c486e-e678-7038-8220-5acab0d57d72 BootID:e78a1a1a-1b7b-4f34-9d4f-b9e608cdb06d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:58:8e:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:58:8e:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:25:07:c0:4a:9c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 20:36:47.688058 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.688057    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 20:36:47.688148 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.688130    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 20:36:47.691022 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.690995    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 20:36:47.691149 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.691025    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-122.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 20:36:47.692620 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.692610    2579 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 20:36:47.692653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.692622    2579 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 20:36:47.692653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.692636    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 20:36:47.692653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.692649    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 20:36:47.693480 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.693471    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 20:36:47.696343 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696332    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 20:36:47.696448 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696439    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 20:36:47.696476 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696468    2579 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 20:36:47.696503 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696481    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 20:36:47.696503 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696492    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 20:36:47.696503 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696500    2579 kubelet.go:398] "Adding apiserver pod source"
Sep 15 20:36:47.696608 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696508    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 20:36:47.696922 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696906    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 20:36:47.696980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.696934    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 20:36:47.700188 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.700167    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 20:36:47.702518 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.702504    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 20:36:47.702574 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.702535    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 20:36:47.704016 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704003    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704020    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704026    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704031    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704039    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704045    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704058    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704063    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704069    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704074    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704083    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 20:36:47.704085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704088    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 20:36:47.704793 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704783    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 20:36:47.704793 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.704794    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 20:36:47.706492 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.706473    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxgj2"
Sep 15 20:36:47.707804 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.707785    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-122.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 20:36:47.707901 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.707801    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-122.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 20:36:47.707983 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.707965    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 20:36:47.708487 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.708473    2579 server.go:1268] "Started kubelet"
Sep 15 20:36:47.708851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.708758    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 20:36:47.709263 ip-10-0-138-122 systemd[1]: Started Kubernetes Kubelet.
Sep 15 20:36:47.709450 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.709303    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 20:36:47.709450 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.709401    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 20:36:47.710691 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.710637    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 20:36:47.711465 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.711451    2579 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 20:36:47.712970 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.712938    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 20:36:47.713302 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.712472    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-122.ec2.internal.18d598f5a2630558  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-122.ec2.internal,UID:ip-10-0-138-122.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-122.ec2.internal,},FirstTimestamp:2026-09-15 20:36:47.708448088 +0000 UTC m=+0.334915981,LastTimestamp:2026-09-15 20:36:47.708448088 +0000 UTC m=+0.334915981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-122.ec2.internal,}"
Sep 15 20:36:47.713457 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.713443    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xxgj2"
Sep 15 20:36:47.716234 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.716221    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 20:36:47.716777 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.716759    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 20:36:47.716880 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.716858    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:47.717333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.717317    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 20:36:47.717450 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.717321    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 20:36:47.717519 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.717453    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 20:36:47.717772 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.717759    2579 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 20:36:47.717772 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.717770    2579 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 20:36:47.718053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718037    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 20:36:47.718102 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718060    2579 factory.go:55] Registering systemd factory
Sep 15 20:36:47.718102 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718070    2579 factory.go:223] Registration of the systemd container factory successfully
Sep 15 20:36:47.718271 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718256    2579 factory.go:153] Registering CRI-O factory
Sep 15 20:36:47.718342 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718272    2579 factory.go:223] Registration of the crio container factory successfully
Sep 15 20:36:47.718342 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718292    2579 factory.go:103] Registering Raw factory
Sep 15 20:36:47.718342 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718305    2579 manager.go:1196] Started watching for new ooms in manager
Sep 15 20:36:47.718744 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.718730    2579 manager.go:349] Starting recovery of all containers
Sep 15 20:36:47.727801 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.727782    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:47.730264 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.730171    2579 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-122.ec2.internal\" not found" node="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.731531 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.731512    2579 manager.go:354] Recovery completed
Sep 15 20:36:47.737238 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.737133    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.739191 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739173    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.739270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739207    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.739270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739222    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.739803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739789    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 20:36:47.739803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739802    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 20:36:47.739944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739810    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.739944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.739822    2579 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 20:36:47.740471 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.740458    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.740544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.740486    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.740544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.740501    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.741022 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.741011    2579 policy_none.go:49] "None policy: Start"
Sep 15 20:36:47.741066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.741026    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 20:36:47.741066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.741036    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 20:36:47.741946 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.741937    2579 policy_none.go:47] "Start"
Sep 15 20:36:47.780259 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.780243    2579 manager.go:337] "Starting Device Plugin manager"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.780283    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.780295    2579 server.go:87] "Starting device plugin registration server"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.780597    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.780611    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.780729    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.781154    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.781198    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.783082    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 20:36:47.791956 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.783429    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:47.820945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.820917    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 20:36:47.822169 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.822144    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 20:36:47.822246 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.822205    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 20:36:47.822246 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.822230    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 20:36:47.822339 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.822311    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 15 20:36:47.824246 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.824227    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:47.881352 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.881320    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.883756 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.883730    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.883838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.883770    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.883838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.883783    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.883838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.883807    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.889316 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.889243    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.889316 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.889266    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-122.ec2.internal\": node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:47.904718 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.904700    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:47.923121 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.923101    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"]
Sep 15 20:36:47.923170 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.923163    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.924597 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.924581    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.924705 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.924608    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.924705 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.924619    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.925876 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.925864    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.926004 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.925992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.926038 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926021    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.926577 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926560    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.926658 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926574    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.926658 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926589    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.926658 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926596    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.926658 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926605    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.926658 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.926607    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.927811 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.927795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.927893 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.927825    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 20:36:47.928763 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.928736    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 20:36:47.928763 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.928758    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 20:36:47.928887 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:47.928772    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeHasSufficientPID"
Sep 15 20:36:47.955223 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.955203    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-122.ec2.internal\" not found" node="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:47.964027 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:47.964013    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-122.ec2.internal\" not found" node="ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.005187 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.005160    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.019296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.019273    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.019377 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.019300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.019377 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.019319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81ca5a9bd4d1a43083fabcf1bcff5aa1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-122.ec2.internal\" (UID: \"81ca5a9bd4d1a43083fabcf1bcff5aa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.105688 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.105640    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.120022 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.119997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.120119 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.120029    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.120119 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.120048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81ca5a9bd4d1a43083fabcf1bcff5aa1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-122.ec2.internal\" (UID: \"81ca5a9bd4d1a43083fabcf1bcff5aa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.120119 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.120100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.120224 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.120109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5dd6e54d4c7aca60ba5b848b3ec87eec-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal\" (UID: \"5dd6e54d4c7aca60ba5b848b3ec87eec\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.120224 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.120142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/81ca5a9bd4d1a43083fabcf1bcff5aa1-config\") pod \"kube-apiserver-proxy-ip-10-0-138-122.ec2.internal\" (UID: \"81ca5a9bd4d1a43083fabcf1bcff5aa1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.206445 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.206365    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.257898 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.257856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.267609 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.267588    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.306597 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.306564    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.407062 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.407031    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.507691 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.507588    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.608147 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.608113    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.655654 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.655624    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 20:36:48.656127 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.655827    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 20:36:48.656127 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.655847    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 20:36:48.708249 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:48.708217    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-122.ec2.internal\" not found"
Sep 15 20:36:48.718745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.718696    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 20:31:49 +0000 UTC" deadline="2028-03-09 17:02:47.8081695 +0000 UTC"
Sep 15 20:36:48.718745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.718738    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12980h25m59.089434683s"
Sep 15 20:36:48.747585 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.747555    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:48.787913 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:48.787874    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd6e54d4c7aca60ba5b848b3ec87eec.slice/crio-c16ac48695aa24a21080534439eade6fea0b5780f032842fd80e522aff9210be WatchSource:0}: Error finding container c16ac48695aa24a21080534439eade6fea0b5780f032842fd80e522aff9210be: Status 404 returned error can't find the container with id c16ac48695aa24a21080534439eade6fea0b5780f032842fd80e522aff9210be
Sep 15 20:36:48.788215 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.788200    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:48.788387 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:48.788369    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ca5a9bd4d1a43083fabcf1bcff5aa1.slice/crio-83a27e9e686bcbd0f29b5dbe4b0a3e9f6066d56f1158aaf3b5e98d096ac9e840 WatchSource:0}: Error finding container 83a27e9e686bcbd0f29b5dbe4b0a3e9f6066d56f1158aaf3b5e98d096ac9e840: Status 404 returned error can't find the container with id 83a27e9e686bcbd0f29b5dbe4b0a3e9f6066d56f1158aaf3b5e98d096ac9e840
Sep 15 20:36:48.792310 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.792252    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 20:36:48.817139 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.817114    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.825501 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.825454    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal" event={"ID":"5dd6e54d4c7aca60ba5b848b3ec87eec","Type":"ContainerStarted","Data":"c16ac48695aa24a21080534439eade6fea0b5780f032842fd80e522aff9210be"}
Sep 15 20:36:48.826624 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.826603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal" event={"ID":"81ca5a9bd4d1a43083fabcf1bcff5aa1","Type":"ContainerStarted","Data":"83a27e9e686bcbd0f29b5dbe4b0a3e9f6066d56f1158aaf3b5e98d096ac9e840"}
Sep 15 20:36:48.827406 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.827390    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 20:36:48.828371 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.828356    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal"
Sep 15 20:36:48.836536 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:48.836521    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 20:36:51.536686 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.536607    2579 apiserver.go:52] "Watching apiserver"
Sep 15 20:36:51.549994 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.549963    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 20:36:51.550368 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.550344    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qwpwz","openshift-multus/multus-dkqxd","openshift-multus/network-metrics-daemon-cflrg","kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal","openshift-dns/node-resolver-mzct6","openshift-image-registry/node-ca-dqclf","openshift-network-diagnostics/network-check-target-fjxt6","openshift-network-operator/iptables-alerter-c4g88","openshift-ovn-kubernetes/ovnkube-node-np6s8","kube-system/global-pull-secret-syncer-lrldg","kube-system/konnectivity-agent-rpv5g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss","openshift-cluster-node-tuning-operator/tuned-pv6bv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal"]
Sep 15 20:36:51.553186 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.553163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.553288 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.553235    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:36:51.555415 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.555390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.557313 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.557291    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 20:36:51.557865 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.557768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.557865 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.557833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:36:51.558887 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.558867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.558887 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.558878    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.559065 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.559046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6cc2t\""
Sep 15 20:36:51.559948 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.559929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.563034 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.562141    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.563034 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.562402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.563034 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.562976    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmvl2\""
Sep 15 20:36:51.563034 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.562991    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 20:36:51.563779 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.562995    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.564695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.564586    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.564900 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.564881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.565115 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.565101    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqk2l\""
Sep 15 20:36:51.565207 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.565138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.566654 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.566638    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.567081 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.567064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.567330 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.567296    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.567739 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.567720    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4p89d\""
Sep 15 20:36:51.567885 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.567867    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 20:36:51.568468 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568452    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysconfig\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568558 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-run\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568558 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-sys\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568558 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-lib-modules\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568558 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568552    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-tuned\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-systemd\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568591    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568605    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-modprobe-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-kubernetes\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-conf\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbndx\" (UniqueName: \"kubernetes.io/projected/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-kube-api-access-fbndx\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-kubelet-config\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-dbus\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-var-lib-kubelet\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-host\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-tmp\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xl4\" (UniqueName: \"kubernetes.io/projected/78f49813-8658-4655-ae9a-8852f2e2eac4-kube-api-access-w2xl4\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.568995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.568971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.569352 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.569335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:51.569423 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.569400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:36:51.569665 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.569648    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.570102 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.569979    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571193    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-69mt2\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571309    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571319    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571341    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571411    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.571851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.571548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.573810 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.573791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.574484 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574432    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 20:36:51.574591 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574512    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.574653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574609    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.574653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574609    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 20:36:51.574717 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574663    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ps6zw\""
Sep 15 20:36:51.574717 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.574680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 20:36:51.575967 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.575953    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-vpqft\""
Sep 15 20:36:51.576159 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.576143    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 20:36:51.576256 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.576211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.576649 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.576633    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 20:36:51.578637 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.578614    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.578745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.578728    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6zwwd\""
Sep 15 20:36:51.578905 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.578885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 20:36:51.579727 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.579703    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 20:36:51.583595 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.583568    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 20:36:51.583820 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.583800    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 20:36:51.583891 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.583800    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsrfz\""
Sep 15 20:36:51.585882 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.585862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 20:36:51.590647 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.590630    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7mvhg"
Sep 15 20:36:51.598175 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.598157    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7mvhg"
Sep 15 20:36:51.658253 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.658226    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 20:36:51.669921 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.669883    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-multus\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.670091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.669921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df934d45-30b6-47d7-a704-ffbdd0b01465-tmp-dir\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.670091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.669950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-systemd-units\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.669964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-netns\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670079    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-bin\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12f64684-4fa9-47db-b5c2-fbd466664932-host\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-slash\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670158    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-config\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-cni-binary-copy\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670219    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-kubernetes\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbndx\" (UniqueName: \"kubernetes.io/projected/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-kube-api-access-fbndx\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.670321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670301    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cnibin\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-log-socket\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-kubernetes\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30b326d6-7647-4372-b6ea-6e8ec9786921-ovn-node-metrics-cert\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-host\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-host\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-node-log\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-var-lib-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670573    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-etc-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670589    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-lib-modules\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-tuned\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.670642 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670636    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-daemon-config\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-agent-certs\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12f64684-4fa9-47db-b5c2-fbd466664932-serviceca\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-lib-modules\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-systemd\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-modprobe-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670829    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-systemd\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-kubelet\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-modprobe-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670913    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df934d45-30b6-47d7-a704-ffbdd0b01465-hosts-file\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670939    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.670968    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-etc-kubernetes\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-device-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671060    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mv4j\" (UniqueName: \"kubernetes.io/projected/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-kube-api-access-5mv4j\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.671292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671163    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-k8s-cni-cncf-io\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysconfig\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-sys\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529fa118-68d7-4532-ae24-c0e7750fcca6-iptables-alerter-script\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671316    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-sys\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671250    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysconfig\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671333    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-hostroot\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-systemd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.671394    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-var-lib-kubelet\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xl4\" (UniqueName: \"kubernetes.io/projected/78f49813-8658-4655-ae9a-8852f2e2eac4-kube-api-access-w2xl4\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-netns\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671502    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-var-lib-kubelet\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.671990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.671574    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.171538417 +0000 UTC m=+2.958662432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.671575    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.671711    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.171699635 +0000 UTC m=+2.958823646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671725    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-kubelet\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-cnibin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-multus-certs\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-netd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671860    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-script-lib\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tczl\" (UniqueName: \"kubernetes.io/projected/30b326d6-7647-4372-b6ea-6e8ec9786921-kube-api-access-8tczl\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-run\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.672894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-socket-dir-parent\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.671986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-ovn\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-env-overrides\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-run\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-system-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-conf\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxj68\" (UniqueName: \"kubernetes.io/projected/6189c3e4-bfa9-4791-a63c-a721871317b3-kube-api-access-xxj68\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-konnectivity-ca\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvfv\" (UniqueName: \"kubernetes.io/projected/529fa118-68d7-4532-ae24-c0e7750fcca6-kube-api-access-fxvfv\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c89v7\" (UniqueName: \"kubernetes.io/projected/df934d45-30b6-47d7-a704-ffbdd0b01465-kube-api-access-c89v7\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-conf\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-kubelet-config\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672250    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-dbus\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672279    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-tmp\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-conf-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672335    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-dbus\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.673633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-system-cni-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672319    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-kubelet-config\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v5gm\" (UniqueName: \"kubernetes.io/projected/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kube-api-access-6v5gm\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529fa118-68d7-4532-ae24-c0e7750fcca6-host-slash\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prlf6\" (UniqueName: \"kubernetes.io/projected/12f64684-4fa9-47db-b5c2-fbd466664932-kube-api-access-prlf6\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-bin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.674469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.672665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-sysctl-d\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.674735 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.674687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-tmp\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.674801 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.674784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/78f49813-8658-4655-ae9a-8852f2e2eac4-etc-tuned\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.684598 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.684577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbndx\" (UniqueName: \"kubernetes.io/projected/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-kube-api-access-fbndx\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:51.684836 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.684804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xl4\" (UniqueName: \"kubernetes.io/projected/78f49813-8658-4655-ae9a-8852f2e2eac4-kube-api-access-w2xl4\") pod \"tuned-pv6bv\" (UID: \"78f49813-8658-4655-ae9a-8852f2e2eac4\") " pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.684940 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.684917    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:51.731589 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.731562    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 20:36:51.773792 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v5gm\" (UniqueName: \"kubernetes.io/projected/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kube-api-access-6v5gm\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.773792 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529fa118-68d7-4532-ae24-c0e7750fcca6-host-slash\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prlf6\" (UniqueName: \"kubernetes.io/projected/12f64684-4fa9-47db-b5c2-fbd466664932-kube-api-access-prlf6\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-bin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-multus\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/529fa118-68d7-4532-ae24-c0e7750fcca6-host-slash\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-multus\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-cni-bin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.774030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.773983    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df934d45-30b6-47d7-a704-ffbdd0b01465-tmp-dir\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-systemd-units\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-netns\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-bin\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12f64684-4fa9-47db-b5c2-fbd466664932-host\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-systemd-units\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774162    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-slash\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774164    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-netns\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-config\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-bin\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/12f64684-4fa9-47db-b5c2-fbd466664932-host\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774212    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-cni-binary-copy\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-slash\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774239    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cnibin\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-log-socket\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30b326d6-7647-4372-b6ea-6e8ec9786921-ovn-node-metrics-cert\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.774331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774317    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df934d45-30b6-47d7-a704-ffbdd0b01465-tmp-dir\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774358    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-node-log\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774368    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-log-socket\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cnibin\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-var-lib-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-etc-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-daemon-config\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-agent-certs\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774567    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-node-log\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12f64684-4fa9-47db-b5c2-fbd466664932-serviceca\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-var-lib-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774616    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-kubelet\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.775276 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df934d45-30b6-47d7-a704-ffbdd0b01465-hosts-file\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774754    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-etc-kubernetes\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-device-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mv4j\" (UniqueName: \"kubernetes.io/projected/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-kube-api-access-5mv4j\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-cni-binary-copy\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774825    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-k8s-cni-cncf-io\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774677    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-config\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-etc-openvswitch\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/12f64684-4fa9-47db-b5c2-fbd466664932-serviceca\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-daemon-config\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-etc-kubernetes\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-k8s-cni-cncf-io\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775191    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-binary-copy\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529fa118-68d7-4532-ae24-c0e7750fcca6-iptables-alerter-script\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.774464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-var-lib-kubelet\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776030 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775268    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-hostroot\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-device-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-systemd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775376    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-registration-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775400    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-hostroot\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/df934d45-30b6-47d7-a704-ffbdd0b01465-hosts-file\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775430    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-etc-selinux\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-systemd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-netns\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-kubelet\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775653    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-cnibin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-multus-certs\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-run-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775691    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-socket-dir\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.776803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775717    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-multus-certs\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-host-run-netns\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-netd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-cnibin\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-kubelet\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-script-lib\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775969    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.775998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tczl\" (UniqueName: \"kubernetes.io/projected/30b326d6-7647-4372-b6ea-6e8ec9786921-kube-api-access-8tczl\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-socket-dir-parent\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-cni-netd\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776247    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-ovn\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-env-overrides\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776302    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-system-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776363    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-socket-dir-parent\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776377    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/529fa118-68d7-4532-ae24-c0e7750fcca6-iptables-alerter-script\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-run-ovn\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776446    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-sys-fs\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776494    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/30b326d6-7647-4372-b6ea-6e8ec9786921-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxj68\" (UniqueName: \"kubernetes.io/projected/6189c3e4-bfa9-4791-a63c-a721871317b3-kube-api-access-xxj68\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-system-cni-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-konnectivity-ca\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvfv\" (UniqueName: \"kubernetes.io/projected/529fa118-68d7-4532-ae24-c0e7750fcca6-kube-api-access-fxvfv\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c89v7\" (UniqueName: \"kubernetes.io/projected/df934d45-30b6-47d7-a704-ffbdd0b01465-kube-api-access-c89v7\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-conf-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776724    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-system-cni-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-env-overrides\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/30b326d6-7647-4372-b6ea-6e8ec9786921-ovnkube-script-lib\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-system-cni-dir\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.777844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.776951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6189c3e4-bfa9-4791-a63c-a721871317b3-multus-conf-dir\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.778433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.777286    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-konnectivity-ca\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.778433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.778103    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a4a6f3b5-90b7-44a5-865b-d27c6dc6254d-agent-certs\") pod \"konnectivity-agent-rpv5g\" (UID: \"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d\") " pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.778433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.778196    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/30b326d6-7647-4372-b6ea-6e8ec9786921-ovn-node-metrics-cert\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.780800 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.780774    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:36:51.780886 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.780802    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:36:51.780886 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.780818    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:51.780973 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:51.780895    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:36:52.280877434 +0000 UTC m=+3.068001449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:51.782153 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.782125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prlf6\" (UniqueName: \"kubernetes.io/projected/12f64684-4fa9-47db-b5c2-fbd466664932-kube-api-access-prlf6\") pod \"node-ca-dqclf\" (UID: \"12f64684-4fa9-47db-b5c2-fbd466664932\") " pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.782278 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.782265    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v5gm\" (UniqueName: \"kubernetes.io/projected/6f9af67d-316a-4245-8dd6-3bb1136ef2d2-kube-api-access-6v5gm\") pod \"aws-ebs-csi-driver-node-h26ss\" (UID: \"6f9af67d-316a-4245-8dd6-3bb1136ef2d2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:51.782791 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.782775    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mv4j\" (UniqueName: \"kubernetes.io/projected/8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05-kube-api-access-5mv4j\") pod \"multus-additional-cni-plugins-qwpwz\" (UID: \"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05\") " pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.787415 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.787360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxj68\" (UniqueName: \"kubernetes.io/projected/6189c3e4-bfa9-4791-a63c-a721871317b3-kube-api-access-xxj68\") pod \"multus-dkqxd\" (UID: \"6189c3e4-bfa9-4791-a63c-a721871317b3\") " pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.788641 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.788594    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tczl\" (UniqueName: \"kubernetes.io/projected/30b326d6-7647-4372-b6ea-6e8ec9786921-kube-api-access-8tczl\") pod \"ovnkube-node-np6s8\" (UID: \"30b326d6-7647-4372-b6ea-6e8ec9786921\") " pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.790256 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.790235    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvfv\" (UniqueName: \"kubernetes.io/projected/529fa118-68d7-4532-ae24-c0e7750fcca6-kube-api-access-fxvfv\") pod \"iptables-alerter-c4g88\" (UID: \"529fa118-68d7-4532-ae24-c0e7750fcca6\") " pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.790338 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.790277    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c89v7\" (UniqueName: \"kubernetes.io/projected/df934d45-30b6-47d7-a704-ffbdd0b01465-kube-api-access-c89v7\") pod \"node-resolver-mzct6\" (UID: \"df934d45-30b6-47d7-a704-ffbdd0b01465\") " pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.866361 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.866327    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pv6bv"
Sep 15 20:36:51.875220 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.875200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-c4g88"
Sep 15 20:36:51.887169 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.887151    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:36:51.890775 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.890759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mzct6"
Sep 15 20:36:51.900139 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.900121    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dqclf"
Sep 15 20:36:51.908730 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.908711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dkqxd"
Sep 15 20:36:51.915345 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.915328    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qwpwz"
Sep 15 20:36:51.922994 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.922978    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:36:51.928634 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:51.928615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss"
Sep 15 20:36:52.179960 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.179920    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:52.180136 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.179967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:52.180136 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.180097    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:52.180235 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.180142    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:52.180235 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.180177    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.180156141 +0000 UTC m=+3.967280171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:52.180235 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.180206    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.180187786 +0000 UTC m=+3.967311811 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:52.381147 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.381116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:52.381311 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.381294    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:36:52.381365 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.381319    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:36:52.381365 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.381329    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:52.381438 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:52.381417    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:36:53.381400298 +0000 UTC m=+4.168524318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:52.449569 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.449536    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf934d45_30b6_47d7_a704_ffbdd0b01465.slice/crio-2b326c88f3ce8696620a713a1f77e46a4e0040ab38183a2618b2e9e63dd372f4 WatchSource:0}: Error finding container 2b326c88f3ce8696620a713a1f77e46a4e0040ab38183a2618b2e9e63dd372f4: Status 404 returned error can't find the container with id 2b326c88f3ce8696620a713a1f77e46a4e0040ab38183a2618b2e9e63dd372f4
Sep 15 20:36:52.450427 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.450332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529fa118_68d7_4532_ae24_c0e7750fcca6.slice/crio-64024f50a42252e00081a4ac2df9f7a0079cfae965e50a7276825228a9561656 WatchSource:0}: Error finding container 64024f50a42252e00081a4ac2df9f7a0079cfae965e50a7276825228a9561656: Status 404 returned error can't find the container with id 64024f50a42252e00081a4ac2df9f7a0079cfae965e50a7276825228a9561656
Sep 15 20:36:52.452698 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.452678    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78f49813_8658_4655_ae9a_8852f2e2eac4.slice/crio-ed9c43a4057eb6924ad9bc3d958fda24628492343c520338b5b5c5fe8b276f30 WatchSource:0}: Error finding container ed9c43a4057eb6924ad9bc3d958fda24628492343c520338b5b5c5fe8b276f30: Status 404 returned error can't find the container with id ed9c43a4057eb6924ad9bc3d958fda24628492343c520338b5b5c5fe8b276f30
Sep 15 20:36:52.453493 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.453464    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6189c3e4_bfa9_4791_a63c_a721871317b3.slice/crio-ea4e0da9587ed7e8f95ba1ae18bb2ff2e85d0fb2156838cf2cca38e3150454c1 WatchSource:0}: Error finding container ea4e0da9587ed7e8f95ba1ae18bb2ff2e85d0fb2156838cf2cca38e3150454c1: Status 404 returned error can't find the container with id ea4e0da9587ed7e8f95ba1ae18bb2ff2e85d0fb2156838cf2cca38e3150454c1
Sep 15 20:36:52.456301 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.456275    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9af67d_316a_4245_8dd6_3bb1136ef2d2.slice/crio-9e776c6d3f84673014acd90936553e64d31ffbf1dc6e3a6aa5f2e36a2a22131e WatchSource:0}: Error finding container 9e776c6d3f84673014acd90936553e64d31ffbf1dc6e3a6aa5f2e36a2a22131e: Status 404 returned error can't find the container with id 9e776c6d3f84673014acd90936553e64d31ffbf1dc6e3a6aa5f2e36a2a22131e
Sep 15 20:36:52.457110 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:36:52.456843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a6f3b5_90b7_44a5_865b_d27c6dc6254d.slice/crio-0dd0db1a841234cc7c126800f6bcbae5d874b4b7b165eccfbfdf543dfb75b27c WatchSource:0}: Error finding container 0dd0db1a841234cc7c126800f6bcbae5d874b4b7b165eccfbfdf543dfb75b27c: Status 404 returned error can't find the container with id 0dd0db1a841234cc7c126800f6bcbae5d874b4b7b165eccfbfdf543dfb75b27c
Sep 15 20:36:52.601980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.601789    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:51 +0000 UTC" deadline="2028-07-02 09:15:37.143434378 +0000 UTC"
Sep 15 20:36:52.601980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.601982    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15732h38m44.541461355s"
Sep 15 20:36:52.671057 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.670996    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal" event={"ID":"81ca5a9bd4d1a43083fabcf1bcff5aa1","Type":"ContainerStarted","Data":"7f18e45d3fc6be8f56da969d4de66f3a100f5065a57a2cecc098b99370b52f51"}
Sep 15 20:36:52.672304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.672283    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqclf" event={"ID":"12f64684-4fa9-47db-b5c2-fbd466664932","Type":"ContainerStarted","Data":"ef73765c56fd280f8d3745ef26c2427050fa1a5d120b99387f76070c46efe0ea"}
Sep 15 20:36:52.673403 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.673383    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerStarted","Data":"310a9aea8a753a8d3e39d843b65d4cec5b872ee4e3f8a43acff407e8285a87ab"}
Sep 15 20:36:52.674661 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.674633    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpv5g" event={"ID":"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d","Type":"ContainerStarted","Data":"0dd0db1a841234cc7c126800f6bcbae5d874b4b7b165eccfbfdf543dfb75b27c"}
Sep 15 20:36:52.675886 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.675860    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss" event={"ID":"6f9af67d-316a-4245-8dd6-3bb1136ef2d2","Type":"ContainerStarted","Data":"9e776c6d3f84673014acd90936553e64d31ffbf1dc6e3a6aa5f2e36a2a22131e"}
Sep 15 20:36:52.677025 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.676991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkqxd" event={"ID":"6189c3e4-bfa9-4791-a63c-a721871317b3","Type":"ContainerStarted","Data":"ea4e0da9587ed7e8f95ba1ae18bb2ff2e85d0fb2156838cf2cca38e3150454c1"}
Sep 15 20:36:52.678117 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.678097    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv6bv" event={"ID":"78f49813-8658-4655-ae9a-8852f2e2eac4","Type":"ContainerStarted","Data":"ed9c43a4057eb6924ad9bc3d958fda24628492343c520338b5b5c5fe8b276f30"}
Sep 15 20:36:52.679280 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.679260    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"48a8fe864ff2293ae4407159aa01f4954f51dec6662486792559a40111d6148d"}
Sep 15 20:36:52.680416 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.680390    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4g88" event={"ID":"529fa118-68d7-4532-ae24-c0e7750fcca6","Type":"ContainerStarted","Data":"64024f50a42252e00081a4ac2df9f7a0079cfae965e50a7276825228a9561656"}
Sep 15 20:36:52.681411 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.681393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mzct6" event={"ID":"df934d45-30b6-47d7-a704-ffbdd0b01465","Type":"ContainerStarted","Data":"2b326c88f3ce8696620a713a1f77e46a4e0040ab38183a2618b2e9e63dd372f4"}
Sep 15 20:36:52.685031 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:52.684958    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-122.ec2.internal" podStartSLOduration=2.684936897 podStartE2EDuration="2.684936897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:52.684410528 +0000 UTC m=+3.471534555" watchObservedRunningTime="2026-09-15 20:36:52.684936897 +0000 UTC m=+3.472060926"
Sep 15 20:36:53.188391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.188291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:53.188391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.188364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:53.188622 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.188519    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:53.188622 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.188584    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:36:55.188565321 +0000 UTC m=+5.975689341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:53.189073 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.189025    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:53.189181 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.189081    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:55.189066156 +0000 UTC m=+5.976190165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:53.389395 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.389357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:53.389584 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.389565    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:36:53.389653 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.389592    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:36:53.389653 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.389605    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:53.389747 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.389670    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:36:55.389652078 +0000 UTC m=+6.176776086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:53.606081 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.605966    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 20:31:51 +0000 UTC" deadline="2028-04-07 09:22:21.99551039 +0000 UTC"
Sep 15 20:36:53.606081 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.606052    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13668h45m28.389463608s"
Sep 15 20:36:53.663176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.663142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:53.663366 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.663289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:36:53.663813 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.663725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:53.663917 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.663833    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:36:53.664033 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.663992    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:53.664113 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:53.664092    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:36:53.706337 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.705158    2579 generic.go:356] "Generic (PLEG): container finished" podID="5dd6e54d4c7aca60ba5b848b3ec87eec" containerID="4542c36d9f947370c2a8b061255d74f478f58284e48d97a2ada55979d4c124b0" exitCode=0
Sep 15 20:36:53.706337 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:53.706080    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal" event={"ID":"5dd6e54d4c7aca60ba5b848b3ec87eec","Type":"ContainerDied","Data":"4542c36d9f947370c2a8b061255d74f478f58284e48d97a2ada55979d4c124b0"}
Sep 15 20:36:54.735345 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:54.735300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal" event={"ID":"5dd6e54d4c7aca60ba5b848b3ec87eec","Type":"ContainerStarted","Data":"c1eae2d6611102fcc1f4a957cc2f1be521b17e6811647fb2950f49e4b7acbb11"}
Sep 15 20:36:55.216101 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.216066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:55.216287 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.216120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:55.216287 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.216277    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:55.216400 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.216334    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:36:59.216314821 +0000 UTC m=+10.003438842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:55.216719 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.216699    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:55.216810 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.216751    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:36:59.216737806 +0000 UTC m=+10.003861827 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:55.417909 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.417322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:55.417909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.417477    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:36:55.417909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.417496    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:36:55.417909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.417508    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:55.417909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.417569    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:36:59.417550395 +0000 UTC m=+10.204674410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:55.664163 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.664122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:55.664334 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.664245    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:36:55.664334 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.664123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:55.664334 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:55.664277    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:55.664505 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.664370    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:36:55.664505 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:55.664460    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:36:57.664371 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:57.664339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:57.664839 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:57.664449    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:36:57.664839 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:57.664743    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:57.664839 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:57.664825    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:36:57.664986 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:57.664855    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:57.664986 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:57.664895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:36:59.248939 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.248898    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:59.249411 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.248956    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:59.249411 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.249105    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:59.249411 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.249124    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:59.249411 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.249187    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:07.249166787 +0000 UTC m=+18.036290807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:36:59.249411 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.249208    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:37:07.249197535 +0000 UTC m=+18.036321549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:36:59.450736 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.450698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:59.450894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.450845    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:36:59.450894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.450868    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:36:59.450894 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.450879    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:59.451161 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.450929    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:37:07.450915473 +0000 UTC m=+18.238039480 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:36:59.665587 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.665557    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:36:59.665752 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.665663    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:36:59.666022 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.665984    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:36:59.666118 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.666097    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:36:59.666175 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:36:59.666160    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:36:59.666256 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:36:59.666239    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:01.662762 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:01.662737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:01.662762 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:01.662764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:01.663207 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:01.662737    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:01.663207 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:01.662836    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:01.663207 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:01.662930    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:01.663207 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:01.662980    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:03.663040 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:03.662853    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:03.663040 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:03.662873    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:03.663040 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:03.662973    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:03.663040 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:03.663001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:03.663590 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:03.663124    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:03.663590 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:03.663217    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:05.662272 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:05.662235    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:05.662272 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:05.662260    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:05.662762 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:05.662373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:05.662762 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:05.662400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:05.662762 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:05.662452    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:05.662762 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:05.662559    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:07.308373 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.308331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:07.308848 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.308385    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:07.308848 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.308532    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:37:07.308848 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.308580    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:37:07.308848 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.308608    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:23.308587158 +0000 UTC m=+34.095711181 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:37:07.308848 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.308634    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:37:23.30861698 +0000 UTC m=+34.095740989 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:37:07.510224 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.510186    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:07.510379 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.510353    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:37:07.510379 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.510375    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:37:07.510489 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.510385    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:37:07.510489 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.510441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:37:23.510423539 +0000 UTC m=+34.297547566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:37:07.662623 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.662589    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:07.662794 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.662633    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:07.662794 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:07.662593    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:07.662794 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.662699    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:07.662955 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.662807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:07.662955 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:07.662934    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:09.663307 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:09.663269    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:09.663713 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:09.663338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:09.663713 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:09.663426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:09.663713 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:09.663486    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:09.663713 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:09.663596    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:09.663713 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:09.663678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:11.662540 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.662239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:11.663078 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.662239    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:11.663078 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:11.662628    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:11.663078 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.662271    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:11.663078 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:11.662696    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:11.663078 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:11.662801    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:11.772643 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.772611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss" event={"ID":"6f9af67d-316a-4245-8dd6-3bb1136ef2d2","Type":"ContainerStarted","Data":"ea1d517ae640d6dceb013e6e96a54ae0fb192a17aa6e3334fa295318fb577876"}
Sep 15 20:37:11.774238 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.774207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dkqxd" event={"ID":"6189c3e4-bfa9-4791-a63c-a721871317b3","Type":"ContainerStarted","Data":"1b5ab5c42f3b480f46d840b67a617a2e586e465c5dc391b1e1f51e7498321c48"}
Sep 15 20:37:11.775986 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.775961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pv6bv" event={"ID":"78f49813-8658-4655-ae9a-8852f2e2eac4","Type":"ContainerStarted","Data":"9df06b80b239439869361a837d14f513a5b89003604c1c45b7fcb9362b69118e"}
Sep 15 20:37:11.779514 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"31811b2dac76c3ff3990a0d12072d82879097fda30c5f07a9b496a3f69692fa6"}
Sep 15 20:37:11.779613 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"4bd3190155b0f798e0a0b1c15fc7172bc9d473d58af1df27a25aa91ac411ef13"}
Sep 15 20:37:11.779613 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"9ed5eaf50460043ecbec4f9225c4354fbc451d040832715b967b204e7ae887c1"}
Sep 15 20:37:11.779725 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779648    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"a37d6a76f228f98d90a8915e36378184492db6462e0506c78114a90260255e92"}
Sep 15 20:37:11.779725 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"af7dfe3f99374b09ae8930f7063296d8af8e3ff177efb8dac9ae826d1ce316d5"}
Sep 15 20:37:11.779819 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.779764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"0a27fe9edb40f21f9b38df50dc5719ccab100b5dba64d9aad5925afafc3dc15e"}
Sep 15 20:37:11.781187 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.781166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mzct6" event={"ID":"df934d45-30b6-47d7-a704-ffbdd0b01465","Type":"ContainerStarted","Data":"aef09cac165a14b44f2af4390d7d07edf847ba4b0f213d45e91f6685431c5419"}
Sep 15 20:37:11.783740 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.783710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dqclf" event={"ID":"12f64684-4fa9-47db-b5c2-fbd466664932","Type":"ContainerStarted","Data":"888913070e59ead6fd182cf0518dd8996932627d631e15972df8c9d7aa46dc39"}
Sep 15 20:37:11.786814 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.786786    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="b1b2baae386246ee4180909e064559362d2c9c4b0d189332e909fc59a8c974ac" exitCode=0
Sep 15 20:37:11.786917 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.786873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"b1b2baae386246ee4180909e064559362d2c9c4b0d189332e909fc59a8c974ac"}
Sep 15 20:37:11.788470 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.788450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rpv5g" event={"ID":"a4a6f3b5-90b7-44a5-865b-d27c6dc6254d","Type":"ContainerStarted","Data":"50c829a4c0c11b76bcf1c001981cb99c7c07c92833fddcc00a38f8be87b5a436"}
Sep 15 20:37:11.791228 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.791193    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-122.ec2.internal" podStartSLOduration=21.791182668 podStartE2EDuration="21.791182668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:36:54.752074306 +0000 UTC m=+5.539198336" watchObservedRunningTime="2026-09-15 20:37:11.791182668 +0000 UTC m=+22.578306697"
Sep 15 20:37:11.791345 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.791316    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dkqxd" podStartSLOduration=4.456097148 podStartE2EDuration="22.79131023s" totalImagesPullingTime="18.335213082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:11.790879716 +0000 UTC m=+22.578003744" watchObservedRunningTime="2026-09-15 20:37:11.79131023 +0000 UTC m=+22.578434259"
Sep 15 20:37:11.805789 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.805756    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pv6bv" podStartSLOduration=4.508359141 podStartE2EDuration="22.805744792s" totalImagesPullingTime="18.297385651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:11.805596753 +0000 UTC m=+22.592720781" watchObservedRunningTime="2026-09-15 20:37:11.805744792 +0000 UTC m=+22.592868821"
Sep 15 20:37:11.821912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.821876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mzct6" podStartSLOduration=4.558285708 podStartE2EDuration="22.82186334s" totalImagesPullingTime="18.263577632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:11.821113529 +0000 UTC m=+22.608237557" watchObservedRunningTime="2026-09-15 20:37:11.82186334 +0000 UTC m=+22.608987368"
Sep 15 20:37:11.834184 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.834147    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dqclf" podStartSLOduration=4.549805153 podStartE2EDuration="22.834136913s" totalImagesPullingTime="18.28433176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:11.83410798 +0000 UTC m=+22.621232010" watchObservedRunningTime="2026-09-15 20:37:11.834136913 +0000 UTC m=+22.621260942"
Sep 15 20:37:11.944457 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:11.944438    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 20:37:12.032296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.032268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:37:12.032836 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.032821    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:37:12.047292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.047247    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rpv5g" podStartSLOduration=9.079902582 podStartE2EDuration="23.047230912s" totalImagesPullingTime="13.96732833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:11.868299526 +0000 UTC m=+22.655423554" watchObservedRunningTime="2026-09-15 20:37:12.047230912 +0000 UTC m=+22.834354940"
Sep 15 20:37:12.633178 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.633061    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T20:37:11.944452632Z","UUID":"c4bfc1fa-433b-4f8a-91ae-1dd0375f2b49","Handler":null,"Name":"","Endpoint":""}
Sep 15 20:37:12.634871 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.634847    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 20:37:12.634986 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.634888    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 20:37:12.794132 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.794048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss" event={"ID":"6f9af67d-316a-4245-8dd6-3bb1136ef2d2","Type":"ContainerStarted","Data":"16279b54814c81bfb0f1518ffcc96a4eaf4c2ded3a74a2d8066b45eac5940eaf"}
Sep 15 20:37:12.795964 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.795935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-c4g88" event={"ID":"529fa118-68d7-4532-ae24-c0e7750fcca6","Type":"ContainerStarted","Data":"64f722e7302ac6b31b27475b99060aed7a6020ec8f7e240a7cfdb555529ba194"}
Sep 15 20:37:12.811993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:12.811943    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-c4g88" podStartSLOduration=5.515234944 podStartE2EDuration="23.811924837s" totalImagesPullingTime="18.296689893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:12.811492055 +0000 UTC m=+23.598616097" watchObservedRunningTime="2026-09-15 20:37:12.811924837 +0000 UTC m=+23.599048865"
Sep 15 20:37:13.662353 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.662315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:13.662521 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.662322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:13.662521 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.662442    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:13.662521 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:13.662433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:13.662686 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:13.662565    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:13.662686 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:13.662667    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:13.801542 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.801457    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss" event={"ID":"6f9af67d-316a-4245-8dd6-3bb1136ef2d2","Type":"ContainerStarted","Data":"eff85d12c48eb625508aa3abbabdc59cc27cfa770dc0aa2898c44658521b440d"}
Sep 15 20:37:13.805508 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.805472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"576008916c183319b7372facfe9febae4a5685bf401a45eb586de4a2d2a66950"}
Sep 15 20:37:13.805663 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.805547    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 20:37:13.820943 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:13.820903    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h26ss" podStartSLOduration=4.567035971 podStartE2EDuration="24.820886741s" totalImagesPullingTime="20.25385077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:13.820856345 +0000 UTC m=+24.607980375" watchObservedRunningTime="2026-09-15 20:37:13.820886741 +0000 UTC m=+24.608010764"
Sep 15 20:37:15.661904 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:15.661867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:15.662362 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:15.661920    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:15.662362 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:15.661991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:15.662362 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:15.662024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:15.662362 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:15.662096    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:15.662362 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:15.662157    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:16.814433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.814210    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="fe4e71a55777429e62fb8633693b257bf798587595cc8e4224bb1086eba6f97d" exitCode=0
Sep 15 20:37:16.814433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.814299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"fe4e71a55777429e62fb8633693b257bf798587595cc8e4224bb1086eba6f97d"}
Sep 15 20:37:16.818534 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.818508    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" event={"ID":"30b326d6-7647-4372-b6ea-6e8ec9786921","Type":"ContainerStarted","Data":"4f894c84bb22cc94ef9dd8c870a8bdd054c6e8d2283e16f0b70c49eb1305eb44"}
Sep 15 20:37:16.818783 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.818766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:16.818890 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.818872    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:16.818960 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.818924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:16.834675 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.834645    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:16.837672 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.837651    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:16.859031 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:16.858961    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8" podStartSLOduration=9.417316052 podStartE2EDuration="27.858949197s" totalImagesPullingTime="18.441633145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:16.857081094 +0000 UTC m=+27.644205125" watchObservedRunningTime="2026-09-15 20:37:16.858949197 +0000 UTC m=+27.646073226"
Sep 15 20:37:17.662193 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:17.662161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:17.662193 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:17.662178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:17.662384 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:17.662161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:17.662384 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:17.662265    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:17.662441 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:17.662374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:17.662486 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:17.662466    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:18.380622 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.380453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjxt6"]
Sep 15 20:37:18.381152 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.380700    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:18.381152 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:18.380804    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:18.383868 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.383844    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lrldg"]
Sep 15 20:37:18.384031 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.383935    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:18.384089 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:18.384062    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:18.384534 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.384514    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cflrg"]
Sep 15 20:37:18.384626 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.384600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:18.384703 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:18.384685    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:18.824562 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.824534    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="115a867f0f728e2eedbe92c056650da9e13cac4064e2603c2b3b39a3ef6a4782" exitCode=0
Sep 15 20:37:18.824704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:18.824615    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"115a867f0f728e2eedbe92c056650da9e13cac4064e2603c2b3b39a3ef6a4782"}
Sep 15 20:37:19.662626 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:19.662594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:19.663216 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:19.662692    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:19.663216 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:19.662713    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:19.663216 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:19.662824    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:19.829553 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:19.829467    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="36a76d7cb7fcdb427e6ef8bc3ab6288f7f762ad7e9094d7d7774b9f3b606a38a" exitCode=0
Sep 15 20:37:19.829553 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:19.829524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"36a76d7cb7fcdb427e6ef8bc3ab6288f7f762ad7e9094d7d7774b9f3b606a38a"}
Sep 15 20:37:20.662600 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:20.662567    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:20.662739 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:20.662680    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:21.038966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:21.038927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:37:21.039148 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:21.039113    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 20:37:21.039854 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:21.039830    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rpv5g"
Sep 15 20:37:21.662259 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:21.662092    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:21.662406 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:21.662102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:21.662448 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:21.662431    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:21.662504 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:21.662324    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:22.662547 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:22.662517    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:22.662908 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:22.662632    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:37:23.323485 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:23.323451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:23.323801 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:23.323496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:23.323801 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.323625    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:37:23.323801 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.323631    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 20:37:23.323801 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.323685    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:37:55.323669001 +0000 UTC m=+66.110793010 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 20:37:23.323801 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.323704    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret podName:a5c75bc2-7d31-4663-a3cd-31b435dcecd4 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:55.323694718 +0000 UTC m=+66.110818730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret") pod "global-pull-secret-syncer-lrldg" (UID: "a5c75bc2-7d31-4663-a3cd-31b435dcecd4") : object "kube-system"/"original-pull-secret" not registered
Sep 15 20:37:23.525984 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:23.525955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:23.526163 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.526107    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 20:37:23.526163 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.526126    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 20:37:23.526163 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.526138    2579 projected.go:196] Error preparing data for projected volume kube-api-access-fzd2n for pod openshift-network-diagnostics/network-check-target-fjxt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:37:23.526294 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.526198    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n podName:13dfe28f-8a9d-42d8-b4df-7c4e232d00cc nodeName:}" failed. No retries permitted until 2026-09-15 20:37:55.526184331 +0000 UTC m=+66.313308350 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fzd2n" (UniqueName: "kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n") pod "network-check-target-fjxt6" (UID: "13dfe28f-8a9d-42d8-b4df-7c4e232d00cc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 20:37:23.662217 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:23.662172    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:23.662418 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:23.662229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:23.662418 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.662329    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lrldg" podUID="a5c75bc2-7d31-4663-a3cd-31b435dcecd4"
Sep 15 20:37:23.664386 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:23.664352    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxt6" podUID="13dfe28f-8a9d-42d8-b4df-7c4e232d00cc"
Sep 15 20:37:24.522660 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.522633    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-122.ec2.internal" event="NodeReady"
Sep 15 20:37:24.522787 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.522775    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 20:37:24.565911 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.563206    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-26p5b"]
Sep 15 20:37:24.578376 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.578211    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-bpdfj"]
Sep 15 20:37:24.578376 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.578345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.581600 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.581270    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 20:37:24.581600 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.581311    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\""
Sep 15 20:37:24.581600 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.581535    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 20:37:24.592142 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.592121    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-26p5b"]
Sep 15 20:37:24.592273 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.592173    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bpdfj"]
Sep 15 20:37:24.592273 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.592252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.595060 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.595034    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-4qdwd\""
Sep 15 20:37:24.595191 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.595117    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 20:37:24.595191 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.595036    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 20:37:24.595191 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.595048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 20:37:24.662158 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.662137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:24.665066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.664996    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 20:37:24.665373 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.665118    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\""
Sep 15 20:37:24.735201 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c29b539-3db1-4827-9171-a832c12afb14-cert\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.735339 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735208    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e43d015-d108-49ce-9e1b-917ea3b35955-config-volume\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.735339 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv92n\" (UniqueName: \"kubernetes.io/projected/3c29b539-3db1-4827-9171-a832c12afb14-kube-api-access-wv92n\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.735339 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.735439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e43d015-d108-49ce-9e1b-917ea3b35955-tmp-dir\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.735439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.735364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82ns\" (UniqueName: \"kubernetes.io/projected/4e43d015-d108-49ce-9e1b-917ea3b35955-kube-api-access-l82ns\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.836492 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c29b539-3db1-4827-9171-a832c12afb14-cert\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.836492 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e43d015-d108-49ce-9e1b-917ea3b35955-config-volume\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.836492 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv92n\" (UniqueName: \"kubernetes.io/projected/3c29b539-3db1-4827-9171-a832c12afb14-kube-api-access-wv92n\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.836646 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.836646 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e43d015-d108-49ce-9e1b-917ea3b35955-tmp-dir\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.836646 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82ns\" (UniqueName: \"kubernetes.io/projected/4e43d015-d108-49ce-9e1b-917ea3b35955-kube-api-access-l82ns\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.836819 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:24.836787    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:24.836939 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:24.836879    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:25.336856185 +0000 UTC m=+36.123980192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:24.837028 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.836941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e43d015-d108-49ce-9e1b-917ea3b35955-tmp-dir\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.837125 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.837104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e43d015-d108-49ce-9e1b-917ea3b35955-config-volume\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.840640 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.840623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3c29b539-3db1-4827-9171-a832c12afb14-cert\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.845067 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.845044    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="41309de6e46101212c390b750cca1617b39e6bf4cdcff85f370631a7175a2219" exitCode=0
Sep 15 20:37:24.845154 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.845100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"41309de6e46101212c390b750cca1617b39e6bf4cdcff85f370631a7175a2219"}
Sep 15 20:37:24.847619 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.847599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82ns\" (UniqueName: \"kubernetes.io/projected/4e43d015-d108-49ce-9e1b-917ea3b35955-kube-api-access-l82ns\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:24.847804 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.847784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv92n\" (UniqueName: \"kubernetes.io/projected/3c29b539-3db1-4827-9171-a832c12afb14-kube-api-access-wv92n\") pod \"ingress-canary-bpdfj\" (UID: \"3c29b539-3db1-4827-9171-a832c12afb14\") " pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:24.900630 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:24.900606    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-bpdfj"
Sep 15 20:37:25.216149 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.216113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-bpdfj"]
Sep 15 20:37:25.219874 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:37:25.219849    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c29b539_3db1_4827_9171_a832c12afb14.slice/crio-401896ec90d5ba8224026b6fc91a237c9a4557e853dc6d10777db1c8bd8924da WatchSource:0}: Error finding container 401896ec90d5ba8224026b6fc91a237c9a4557e853dc6d10777db1c8bd8924da: Status 404 returned error can't find the container with id 401896ec90d5ba8224026b6fc91a237c9a4557e853dc6d10777db1c8bd8924da
Sep 15 20:37:25.340840 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.340810    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:25.340997 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:25.340939    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:25.340997 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:25.340990    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:26.340976943 +0000 UTC m=+37.128100953 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:25.662549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.662513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:25.662718 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.662692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:25.665500 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.665428    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\""
Sep 15 20:37:25.665500 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.665470    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 20:37:25.665500 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.665436    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 20:37:25.665955 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.665509    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 20:37:25.848629 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.848593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bpdfj" event={"ID":"3c29b539-3db1-4827-9171-a832c12afb14","Type":"ContainerStarted","Data":"401896ec90d5ba8224026b6fc91a237c9a4557e853dc6d10777db1c8bd8924da"}
Sep 15 20:37:25.852320 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.852292    2579 generic.go:356] "Generic (PLEG): container finished" podID="8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05" containerID="01bbcd4271e31640ec3d5f962b817023a1dddd7cf077174311f63dfca055fff8" exitCode=0
Sep 15 20:37:25.852434 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:25.852336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerDied","Data":"01bbcd4271e31640ec3d5f962b817023a1dddd7cf077174311f63dfca055fff8"}
Sep 15 20:37:26.348611 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:26.348521    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:26.348776 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:26.348702    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:26.348776 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:26.348772    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:28.348754013 +0000 UTC m=+39.135878023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:26.858352 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:26.858319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" event={"ID":"8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05","Type":"ContainerStarted","Data":"8abe4a739978ba178327b9c088f843c1a336ad6dbf2493cb992a0cc54490dce5"}
Sep 15 20:37:26.880834 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:26.880779    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qwpwz" podStartSLOduration=8.535040351 podStartE2EDuration="37.88076465s" totalImagesPullingTime="29.345724299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:26.878674213 +0000 UTC m=+37.665798252" watchObservedRunningTime="2026-09-15 20:37:26.88076465 +0000 UTC m=+37.667888676"
Sep 15 20:37:27.862663 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:27.862625    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-bpdfj" event={"ID":"3c29b539-3db1-4827-9171-a832c12afb14","Type":"ContainerStarted","Data":"8e20beb1e0fff99657abf53038831ac1561b9f72217c5c1fb3bfca44a8407c48"}
Sep 15 20:37:27.878158 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:27.878111    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-bpdfj" podStartSLOduration=1.383931593 podStartE2EDuration="3.87809628s" totalImagesPullingTime="2.494164687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:37:27.877812063 +0000 UTC m=+38.664936092" watchObservedRunningTime="2026-09-15 20:37:27.87809628 +0000 UTC m=+38.665220350"
Sep 15 20:37:28.363566 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:28.363516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:28.363730 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:28.363669    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:28.363772 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:28.363736    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:32.363717999 +0000 UTC m=+43.150842025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:32.389415 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:32.389367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:32.389868 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:32.389513    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:32.389868 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:32.389577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:40.389560979 +0000 UTC m=+51.176685015 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:40.448974 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:40.448931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:40.449450 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:40.449073    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:40.449450 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:40.449125    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:37:56.449111336 +0000 UTC m=+67.236235355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:48.835350 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:48.835322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-np6s8"
Sep 15 20:37:55.358425 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.358391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:55.358425 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.358438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:37:55.361070 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.361046    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 20:37:55.361278 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.361262    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 20:37:55.369159 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:55.369135    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 20:37:55.369229 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:55.369218    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:38:59.369195915 +0000 UTC m=+130.156319941 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : secret "metrics-daemon-secret" not found
Sep 15 20:37:55.372302 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.372284    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a5c75bc2-7d31-4663-a3cd-31b435dcecd4-original-pull-secret\") pod \"global-pull-secret-syncer-lrldg\" (UID: \"a5c75bc2-7d31-4663-a3cd-31b435dcecd4\") " pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:55.558982 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.558945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:55.561919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.561902    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 20:37:55.572526 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.572510    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 20:37:55.583106 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.583078    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzd2n\" (UniqueName: \"kubernetes.io/projected/13dfe28f-8a9d-42d8-b4df-7c4e232d00cc-kube-api-access-fzd2n\") pod \"network-check-target-fjxt6\" (UID: \"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc\") " pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:55.671690 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.671663    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lrldg"
Sep 15 20:37:55.679287 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.679267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zzk8x\""
Sep 15 20:37:55.687349 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.687308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:37:55.977812 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.977713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lrldg"]
Sep 15 20:37:55.981544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:55.981513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjxt6"]
Sep 15 20:37:55.982713 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:37:55.982687    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5c75bc2_7d31_4663_a3cd_31b435dcecd4.slice/crio-b0d0e4dc559a1c7fbd9599e58ba9ece3e903930eee1fb4db6fb76ce369b0d3e5 WatchSource:0}: Error finding container b0d0e4dc559a1c7fbd9599e58ba9ece3e903930eee1fb4db6fb76ce369b0d3e5: Status 404 returned error can't find the container with id b0d0e4dc559a1c7fbd9599e58ba9ece3e903930eee1fb4db6fb76ce369b0d3e5
Sep 15 20:37:55.985114 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:37:55.985076    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13dfe28f_8a9d_42d8_b4df_7c4e232d00cc.slice/crio-f8440d5b531d287d4a68608f4ec1ebe66aa8fdc63041c16beac40a8faa853ea3 WatchSource:0}: Error finding container f8440d5b531d287d4a68608f4ec1ebe66aa8fdc63041c16beac40a8faa853ea3: Status 404 returned error can't find the container with id f8440d5b531d287d4a68608f4ec1ebe66aa8fdc63041c16beac40a8faa853ea3
Sep 15 20:37:56.465441 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:56.465402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:37:56.465806 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:56.465562    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:37:56.465806 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:37:56.465640    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:38:28.465623577 +0000 UTC m=+99.252747597 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:37:56.933786 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:56.933674    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjxt6" event={"ID":"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc","Type":"ContainerStarted","Data":"f8440d5b531d287d4a68608f4ec1ebe66aa8fdc63041c16beac40a8faa853ea3"}
Sep 15 20:37:56.936131 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:37:56.936099    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lrldg" event={"ID":"a5c75bc2-7d31-4663-a3cd-31b435dcecd4","Type":"ContainerStarted","Data":"b0d0e4dc559a1c7fbd9599e58ba9ece3e903930eee1fb4db6fb76ce369b0d3e5"}
Sep 15 20:38:00.948462 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:00.948379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjxt6" event={"ID":"13dfe28f-8a9d-42d8-b4df-7c4e232d00cc","Type":"ContainerStarted","Data":"69e9a3e9c1b8f8e80229426b88b36d71ce8d54846d83b14872e214ac7188c033"}
Sep 15 20:38:00.948906 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:00.948504    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:38:00.949920 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:00.949898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lrldg" event={"ID":"a5c75bc2-7d31-4663-a3cd-31b435dcecd4","Type":"ContainerStarted","Data":"9485f4e999514ba80445083c9ab8685f9eb662469b178828f20ecc313c0a4b5c"}
Sep 15 20:38:00.963275 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:00.963238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fjxt6" podStartSLOduration=67.254678883 podStartE2EDuration="1m11.963225887s" totalImagesPullingTime="4.708547004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:00.962061833 +0000 UTC m=+71.749185862" watchObservedRunningTime="2026-09-15 20:38:00.963225887 +0000 UTC m=+71.750349914"
Sep 15 20:38:00.975423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:00.975388    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lrldg" podStartSLOduration=67.256361418 podStartE2EDuration="1m11.975377701s" totalImagesPullingTime="4.719016283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:00.974970082 +0000 UTC m=+71.762094111" watchObservedRunningTime="2026-09-15 20:38:00.975377701 +0000 UTC m=+71.762501750"
Sep 15 20:38:01.615243 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.615207    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"]
Sep 15 20:38:01.617283 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.617267    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.619944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.619924    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 20:38:01.620088 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.619960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 15 20:38:01.620088 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.619961    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 15 20:38:01.621141 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.621115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 15 20:38:01.621141 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.621137    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 20:38:01.621313 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.621172    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 15 20:38:01.621313 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.621232    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 20:38:01.629840 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.629820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"]
Sep 15 20:38:01.701725 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.701725 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.701945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtc4q\" (UniqueName: \"kubernetes.io/projected/acf24153-a5fc-4b79-9815-ef87af90518b-kube-api-access-jtc4q\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.701945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.701945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.701945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.701881    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf24153-a5fc-4b79-9815-ef87af90518b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802130 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802236 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf24153-a5fc-4b79-9815-ef87af90518b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802236 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802195    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802236 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802213    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802236 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtc4q\" (UniqueName: \"kubernetes.io/projected/acf24153-a5fc-4b79-9815-ef87af90518b-kube-api-access-jtc4q\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802398 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.802982 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.802959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf24153-a5fc-4b79-9815-ef87af90518b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.804719 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.804698    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.804926 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.804903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.805216 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.805198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-ca\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.805296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.805228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/acf24153-a5fc-4b79-9815-ef87af90518b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.810397 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.810378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtc4q\" (UniqueName: \"kubernetes.io/projected/acf24153-a5fc-4b79-9815-ef87af90518b-kube-api-access-jtc4q\") pod \"cluster-proxy-proxy-agent-d4fdd86f6-pzhw7\" (UID: \"acf24153-a5fc-4b79-9815-ef87af90518b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:01.945635 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:01.945596    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:38:02.215202 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:02.215127    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"]
Sep 15 20:38:02.218181 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:38:02.218150    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podacf24153_a5fc_4b79_9815_ef87af90518b.slice/crio-ce5afd61d52c46d4cd44f3c0b3e7929e1801798fbfe6ef2820e5960791c459fd WatchSource:0}: Error finding container ce5afd61d52c46d4cd44f3c0b3e7929e1801798fbfe6ef2820e5960791c459fd: Status 404 returned error can't find the container with id ce5afd61d52c46d4cd44f3c0b3e7929e1801798fbfe6ef2820e5960791c459fd
Sep 15 20:38:02.956616 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:02.956583    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerStarted","Data":"ce5afd61d52c46d4cd44f3c0b3e7929e1801798fbfe6ef2820e5960791c459fd"}
Sep 15 20:38:06.972885 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:06.972846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerStarted","Data":"88aac820a52c0aa1ad80c3c5144506d89c6331b8607c9f3653253357cef3ce6f"}
Sep 15 20:38:06.972885 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:06.972894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerStarted","Data":"5b8f970d8f283e4f68187cfc906e2bae4d64bdf133be6357155babb3fbc996bd"}
Sep 15 20:38:06.973359 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:06.972908    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerStarted","Data":"ea3b2f4f2c788e49968b4af733639f70d355c6443a67e5a0d0bf83d52a0f4ff2"}
Sep 15 20:38:06.993368 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:06.993322    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" podStartSLOduration=2.177383232 podStartE2EDuration="5.993288743s" totalImagesPullingTime="3.815905511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:38:06.993268527 +0000 UTC m=+77.780392557" watchObservedRunningTime="2026-09-15 20:38:06.993288743 +0000 UTC m=+77.780412764"
Sep 15 20:38:28.489293 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:28.489246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:38:28.489711 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:38:28.489393    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:38:28.489711 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:38:28.489460    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:39:32.489444595 +0000 UTC m=+163.276568606 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:38:31.954745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:31.954717    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fjxt6"
Sep 15 20:38:59.396387 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:38:59.396343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:38:59.396854 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:38:59.396455    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 20:38:59.396854 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:38:59.396520    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:41:01.39650426 +0000 UTC m=+252.183628267 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : secret "metrics-daemon-secret" not found
Sep 15 20:39:27.589433 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:39:27.589373    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-26p5b" podUID="4e43d015-d108-49ce-9e1b-917ea3b35955"
Sep 15 20:39:27.712285 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:39:27.712250    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:39:28.200879 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:39:28.200803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:39:32.520027 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:39:32.519963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:39:32.520439 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:39:32.520118    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 20:39:32.520439 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:39:32.520196    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls podName:4e43d015-d108-49ce-9e1b-917ea3b35955 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:34.520177817 +0000 UTC m=+285.307301845 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls") pod "dns-default-26p5b" (UID: "4e43d015-d108-49ce-9e1b-917ea3b35955") : secret "dns-default-metrics-tls" not found
Sep 15 20:39:41.662553 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:39:41.662470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:41:01.432730 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:01.432678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:41:01.433234 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:01.432833    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 15 20:41:01.433234 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:01.432911    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs podName:2dc71d14-d23a-437d-ba3d-1edf2ffb7dde nodeName:}" failed. No retries permitted until 2026-09-15 20:43:03.432893527 +0000 UTC m=+374.220017534 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs") pod "network-metrics-daemon-cflrg" (UID: "2dc71d14-d23a-437d-ba3d-1edf2ffb7dde") : secret "metrics-daemon-secret" not found
Sep 15 20:41:05.202876 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.202838    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x87rb"]
Sep 15 20:41:05.205199 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.205178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.208018 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.207976    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 20:41:05.208131 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.208060    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 20:41:05.208131 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.207980    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 20:41:05.209239 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.209219    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d898r\""
Sep 15 20:41:05.209333 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.209223    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 20:41:05.218761 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.218729    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x87rb"]
Sep 15 20:41:05.358341 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.358302    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.358341 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.358341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae55e282-d1ba-4200-bb42-620685f1b1ee-data-volume\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.358544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.358360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae55e282-d1ba-4200-bb42-620685f1b1ee-crio-socket\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.358544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.358375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml99p\" (UniqueName: \"kubernetes.io/projected/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-api-access-ml99p\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.358544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.358456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.458936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae55e282-d1ba-4200-bb42-620685f1b1ee-data-volume\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.458980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae55e282-d1ba-4200-bb42-620685f1b1ee-crio-socket\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.459000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml99p\" (UniqueName: \"kubernetes.io/projected/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-api-access-ml99p\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.459065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.459094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.459104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ae55e282-d1ba-4200-bb42-620685f1b1ee-crio-socket\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.459176    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.459181    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.459224    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.959211771 +0000 UTC m=+256.746335798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:05.459285 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.459237    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:05.959230441 +0000 UTC m=+256.746354448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : configmap "kube-rbac-proxy" not found
Sep 15 20:41:05.459529 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.459337    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ae55e282-d1ba-4200-bb42-620685f1b1ee-data-volume\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.467150 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.467123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml99p\" (UniqueName: \"kubernetes.io/projected/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-api-access-ml99p\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.962324 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.962290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.962490 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:05.962344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:05.962490 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.962413    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:41:05.962490 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.962470    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:05.962652 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.962500    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.962478652 +0000 UTC m=+257.749602660 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : configmap "kube-rbac-proxy" not found
Sep 15 20:41:05.962652 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:05.962518    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:06.962507636 +0000 UTC m=+257.749631664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:06.969158 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:06.969114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:06.969679 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:06.969178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:06.969679 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:06.969229    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:41:06.969679 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:06.969305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.969288896 +0000 UTC m=+259.756412920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : configmap "kube-rbac-proxy" not found
Sep 15 20:41:06.969679 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:06.969305    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:06.969679 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:06.969357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:08.969342781 +0000 UTC m=+259.756466807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:07.330187 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.330116    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mzct6_df934d45-30b6-47d7-a704-ffbdd0b01465/dns-node-resolver/0.log"
Sep 15 20:41:07.919419 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.919389    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pv5j7"]
Sep 15 20:41:07.921421 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.921406    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:07.923766 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.923739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-mpmr2\""
Sep 15 20:41:07.923961 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.923946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 15 20:41:07.924927 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.924912    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 15 20:41:07.925045 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.924927    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 15 20:41:07.925045 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.924970    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 15 20:41:07.929253 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:07.929230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pv5j7"]
Sep 15 20:41:08.076833 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.076783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pz7d\" (UniqueName: \"kubernetes.io/projected/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-kube-api-access-6pz7d\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.077274 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.076856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-cabundle\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.077274 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.076890    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-key\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.130571 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.130545    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqclf_12f64684-4fa9-47db-b5c2-fbd466664932/node-ca/0.log"
Sep 15 20:41:08.177384 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.177305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-key\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.177494 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.177390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pz7d\" (UniqueName: \"kubernetes.io/projected/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-kube-api-access-6pz7d\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.177494 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.177431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-cabundle\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.178060 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.178041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-cabundle\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.179815 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.179798    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-signing-key\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.185800 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.185776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pz7d\" (UniqueName: \"kubernetes.io/projected/cb1cf214-a843-4c5c-8cc8-3823eb7b399e-kube-api-access-6pz7d\") pod \"service-ca-568869fdc5-pv5j7\" (UID: \"cb1cf214-a843-4c5c-8cc8-3823eb7b399e\") " pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.230915 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.230882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-pv5j7"
Sep 15 20:41:08.507477 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.507409    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-pv5j7"]
Sep 15 20:41:08.510581 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:08.510556    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb1cf214_a843_4c5c_8cc8_3823eb7b399e.slice/crio-c3ebffb87282e9e5dbc0e3351831c7b1bda9b53efaa3e8ae9d70bde7b3a92000 WatchSource:0}: Error finding container c3ebffb87282e9e5dbc0e3351831c7b1bda9b53efaa3e8ae9d70bde7b3a92000: Status 404 returned error can't find the container with id c3ebffb87282e9e5dbc0e3351831c7b1bda9b53efaa3e8ae9d70bde7b3a92000
Sep 15 20:41:08.730110 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.730088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bpdfj_3c29b539-3db1-4827-9171-a832c12afb14/serve-healthcheck-canary/0.log"
Sep 15 20:41:08.983834 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.983795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:08.984063 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:08.983853    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:08.984063 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:08.983922    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:41:08.984063 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:08.983971    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:08.984063 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:08.983999    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.983981197 +0000 UTC m=+263.771105216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : configmap "kube-rbac-proxy" not found
Sep 15 20:41:08.984063 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:08.984047    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:12.984027437 +0000 UTC m=+263.771151461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:09.476591 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:09.476550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-pv5j7" event={"ID":"cb1cf214-a843-4c5c-8cc8-3823eb7b399e","Type":"ContainerStarted","Data":"c3ebffb87282e9e5dbc0e3351831c7b1bda9b53efaa3e8ae9d70bde7b3a92000"}
Sep 15 20:41:11.484193 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:11.484108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-pv5j7" event={"ID":"cb1cf214-a843-4c5c-8cc8-3823eb7b399e","Type":"ContainerStarted","Data":"d1340d681fe10512a9ced7f0789647debae916e561c989b0f9c4864fed211c4c"}
Sep 15 20:41:11.500339 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:11.500291    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-pv5j7" podStartSLOduration=2.486829212 podStartE2EDuration="4.500276106s" totalImagesPullingTime="2.013446894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:11.499397598 +0000 UTC m=+262.286521618" watchObservedRunningTime="2026-09-15 20:41:11.500276106 +0000 UTC m=+262.287400134"
Sep 15 20:41:13.014405 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:13.014368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:13.014909 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:13.014459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:13.014909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:13.014520    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:13.014909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:13.014529    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 20:41:13.014909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:13.014588    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:21.01457067 +0000 UTC m=+271.801694683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:13.014909 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:13.014604    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:21.014596441 +0000 UTC m=+271.801720448 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : configmap "kube-rbac-proxy" not found
Sep 15 20:41:21.073377 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:21.073333    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:21.073858 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:21.073396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:21.073858 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:21.073518    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:21.073858 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:21.073612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls podName:ae55e282-d1ba-4200-bb42-620685f1b1ee nodeName:}" failed. No retries permitted until 2026-09-15 20:41:37.073591212 +0000 UTC m=+287.860715240 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-x87rb" (UID: "ae55e282-d1ba-4200-bb42-620685f1b1ee") : secret "insights-runtime-extractor-tls" not found
Sep 15 20:41:21.074049 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:21.073857    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ae55e282-d1ba-4200-bb42-620685f1b1ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:28.881644 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.881608    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"]
Sep 15 20:41:28.887025 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.886988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:28.889897 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.889877    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 20:41:28.890046 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.889900    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 20:41:28.890046 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.889902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 20:41:28.890046 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.889988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xgtm7\""
Sep 15 20:41:28.895535 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.895514    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 20:41:28.898146 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:28.898128    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"]
Sep 15 20:41:29.030311 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030275    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-certificates\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-installation-pull-secrets\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-tls\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030392    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-image-registry-private-configuration\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2304952d-987b-4c91-923d-8d3a1f55fcbd-ca-trust-extracted\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-trusted-ca\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030483 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5vlm\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-kube-api-access-f5vlm\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.030690 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.030497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-bound-sa-token\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131738 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5vlm\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-kube-api-access-f5vlm\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131738 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-bound-sa-token\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-certificates\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-installation-pull-secrets\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-tls\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131855    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-image-registry-private-configuration\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131889    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2304952d-987b-4c91-923d-8d3a1f55fcbd-ca-trust-extracted\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.131954 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.131911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-trusted-ca\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.132427 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.132395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2304952d-987b-4c91-923d-8d3a1f55fcbd-ca-trust-extracted\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.132815 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.132794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-trusted-ca\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.132903 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.132861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-certificates\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.134456 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.134425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-installation-pull-secrets\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.134545 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.134487    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2304952d-987b-4c91-923d-8d3a1f55fcbd-image-registry-private-configuration\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.134631 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.134613    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-registry-tls\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.139639 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.139617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-bound-sa-token\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.140023 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.139985    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5vlm\" (UniqueName: \"kubernetes.io/projected/2304952d-987b-4c91-923d-8d3a1f55fcbd-kube-api-access-f5vlm\") pod \"image-registry-6b69dc5c9c-2dkxx\" (UID: \"2304952d-987b-4c91-923d-8d3a1f55fcbd\") " pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.195545 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.195518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:29.470271 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.470196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"]
Sep 15 20:41:29.473157 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:29.473128    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2304952d_987b_4c91_923d_8d3a1f55fcbd.slice/crio-caf35e1092c26d2a32ebf19fbdadb63b911c74c10905f7d2b1c21fe4dcf45e16 WatchSource:0}: Error finding container caf35e1092c26d2a32ebf19fbdadb63b911c74c10905f7d2b1c21fe4dcf45e16: Status 404 returned error can't find the container with id caf35e1092c26d2a32ebf19fbdadb63b911c74c10905f7d2b1c21fe4dcf45e16
Sep 15 20:41:29.535730 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:29.535699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx" event={"ID":"2304952d-987b-4c91-923d-8d3a1f55fcbd","Type":"ContainerStarted","Data":"caf35e1092c26d2a32ebf19fbdadb63b911c74c10905f7d2b1c21fe4dcf45e16"}
Sep 15 20:41:30.539732 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:30.539698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx" event={"ID":"2304952d-987b-4c91-923d-8d3a1f55fcbd","Type":"ContainerStarted","Data":"33474bc760fc4e3afdb948054c175a6be0e2e58a35a171ec48a221c679347e96"}
Sep 15 20:41:30.540127 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:30.539822    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:30.561164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:30.561121    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx" podStartSLOduration=2.561106931 podStartE2EDuration="2.561106931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:30.559280393 +0000 UTC m=+281.346404423" watchObservedRunningTime="2026-09-15 20:41:30.561106931 +0000 UTC m=+281.348230959"
Sep 15 20:41:31.201344 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:31.201302    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-26p5b" podUID="4e43d015-d108-49ce-9e1b-917ea3b35955"
Sep 15 20:41:31.542917 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:31.542833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:34.569178 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:34.569149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:34.571577 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:34.571553    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e43d015-d108-49ce-9e1b-917ea3b35955-metrics-tls\") pod \"dns-default-26p5b\" (UID: \"4e43d015-d108-49ce-9e1b-917ea3b35955\") " pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:34.846416 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:34.846331    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lx25c\""
Sep 15 20:41:34.854456 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:34.854426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:35.133226 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:35.133194    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-26p5b"]
Sep 15 20:41:35.136923 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:35.136899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e43d015_d108_49ce_9e1b_917ea3b35955.slice/crio-1f563e52b07e44f130f09611f1ea39a4fd6fae97b74d6ebbfa8d19c7c0c290a3 WatchSource:0}: Error finding container 1f563e52b07e44f130f09611f1ea39a4fd6fae97b74d6ebbfa8d19c7c0c290a3: Status 404 returned error can't find the container with id 1f563e52b07e44f130f09611f1ea39a4fd6fae97b74d6ebbfa8d19c7c0c290a3
Sep 15 20:41:35.555262 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:35.555172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26p5b" event={"ID":"4e43d015-d108-49ce-9e1b-917ea3b35955","Type":"ContainerStarted","Data":"1f563e52b07e44f130f09611f1ea39a4fd6fae97b74d6ebbfa8d19c7c0c290a3"}
Sep 15 20:41:37.087331 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.087289    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:37.090022 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.089973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ae55e282-d1ba-4200-bb42-620685f1b1ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x87rb\" (UID: \"ae55e282-d1ba-4200-bb42-620685f1b1ee\") " pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:37.314403 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.314363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x87rb"
Sep 15 20:41:37.563612 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.563587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26p5b" event={"ID":"4e43d015-d108-49ce-9e1b-917ea3b35955","Type":"ContainerStarted","Data":"4dc38f1cbc14eccda10452ec1940ca3f7cbc9098d2e8b6e688b6b8651f623bff"}
Sep 15 20:41:37.604185 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.604105    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x87rb"]
Sep 15 20:41:37.606865 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:37.606837    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae55e282_d1ba_4200_bb42_620685f1b1ee.slice/crio-8ca6e9847d564ea19b05677dfd91d2c5c0c0b94dc73319b81f5ca2753be07ba8 WatchSource:0}: Error finding container 8ca6e9847d564ea19b05677dfd91d2c5c0c0b94dc73319b81f5ca2753be07ba8: Status 404 returned error can't find the container with id 8ca6e9847d564ea19b05677dfd91d2c5c0c0b94dc73319b81f5ca2753be07ba8
Sep 15 20:41:37.997395 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:37.997360    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"]
Sep 15 20:41:38.000281 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.000266    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.003118 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.003090    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 15 20:41:38.003251 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.003132    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 20:41:38.004186 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.004166    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 20:41:38.004364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.004349    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 20:41:38.004429 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.004373    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 15 20:41:38.004694 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.004676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2qmk6\""
Sep 15 20:41:38.008510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.008489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"]
Sep 15 20:41:38.094661 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.094623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.095032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.094698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.095032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.094749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.095032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.094776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csf8v\" (UniqueName: \"kubernetes.io/projected/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-kube-api-access-csf8v\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.195406 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.195371    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.195559 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.195424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.195559 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.195462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.195559 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.195479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csf8v\" (UniqueName: \"kubernetes.io/projected/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-kube-api-access-csf8v\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.195559 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:38.195532    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 20:41:38.195689 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:38.195614    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls podName:0dccda17-f48c-46c1-bcba-ea7e6b6bd355 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:38.695596769 +0000 UTC m=+289.482720794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-cdlks" (UID: "0dccda17-f48c-46c1-bcba-ea7e6b6bd355") : secret "prometheus-operator-tls" not found
Sep 15 20:41:38.196299 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.196273    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.197971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.197948    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.204529 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.204505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csf8v\" (UniqueName: \"kubernetes.io/projected/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-kube-api-access-csf8v\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.568884 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.568845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x87rb" event={"ID":"ae55e282-d1ba-4200-bb42-620685f1b1ee","Type":"ContainerStarted","Data":"2d96e926d691513705f0d3d64a8e99c12252707e324be9d541c8bd4df010ac13"}
Sep 15 20:41:38.569047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.568924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x87rb" event={"ID":"ae55e282-d1ba-4200-bb42-620685f1b1ee","Type":"ContainerStarted","Data":"28efa2e65263eaee6b6dec68f6bea8b1763c39adb75b688efb21c55c47ddf0ee"}
Sep 15 20:41:38.569047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.568955    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x87rb" event={"ID":"ae55e282-d1ba-4200-bb42-620685f1b1ee","Type":"ContainerStarted","Data":"8ca6e9847d564ea19b05677dfd91d2c5c0c0b94dc73319b81f5ca2753be07ba8"}
Sep 15 20:41:38.570732 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.570708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-26p5b" event={"ID":"4e43d015-d108-49ce-9e1b-917ea3b35955","Type":"ContainerStarted","Data":"5811f5aa3f39ebe53f24f98834ad51ecf570587bf956ca3c8d90f831e881fc7f"}
Sep 15 20:41:38.570856 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.570844    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:38.588170 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.588122    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-26p5b" podStartSLOduration=252.337768642 podStartE2EDuration="4m14.588107874s" totalImagesPullingTime="2.250339232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:37:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:38.586666173 +0000 UTC m=+289.373790203" watchObservedRunningTime="2026-09-15 20:41:38.588107874 +0000 UTC m=+289.375231952"
Sep 15 20:41:38.699650 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:38.699564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:38.699794 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:38.699687    2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Sep 15 20:41:38.699794 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:38.699738    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls podName:0dccda17-f48c-46c1-bcba-ea7e6b6bd355 nodeName:}" failed. No retries permitted until 2026-09-15 20:41:39.699725914 +0000 UTC m=+290.486849926 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-cdlks" (UID: "0dccda17-f48c-46c1-bcba-ea7e6b6bd355") : secret "prometheus-operator-tls" not found
Sep 15 20:41:39.707630 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:39.707592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:39.710648 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:39.710612    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/0dccda17-f48c-46c1-bcba-ea7e6b6bd355-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cdlks\" (UID: \"0dccda17-f48c-46c1-bcba-ea7e6b6bd355\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:39.809704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:39.809672    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"
Sep 15 20:41:40.080297 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:40.080216    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cdlks"]
Sep 15 20:41:40.084360 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:40.084330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dccda17_f48c_46c1_bcba_ea7e6b6bd355.slice/crio-7ec7461141f43e3b2fecfd37b1d5b43b7e565846c469a30151c009b0915a8bf2 WatchSource:0}: Error finding container 7ec7461141f43e3b2fecfd37b1d5b43b7e565846c469a30151c009b0915a8bf2: Status 404 returned error can't find the container with id 7ec7461141f43e3b2fecfd37b1d5b43b7e565846c469a30151c009b0915a8bf2
Sep 15 20:41:40.578535 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:40.578501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks" event={"ID":"0dccda17-f48c-46c1-bcba-ea7e6b6bd355","Type":"ContainerStarted","Data":"7ec7461141f43e3b2fecfd37b1d5b43b7e565846c469a30151c009b0915a8bf2"}
Sep 15 20:41:41.584057 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:41.584018    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x87rb" event={"ID":"ae55e282-d1ba-4200-bb42-620685f1b1ee","Type":"ContainerStarted","Data":"ec8df5dd2bf7ecdbd120dc39d06d476ad008712b6ac1879d6d08cd4064c7e8f4"}
Sep 15 20:41:41.602881 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:41.602836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x87rb" podStartSLOduration=33.780986203 podStartE2EDuration="36.602820639s" totalImagesPullingTime="2.821834436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:41.602588604 +0000 UTC m=+292.389712632" watchObservedRunningTime="2026-09-15 20:41:41.602820639 +0000 UTC m=+292.389944667"
Sep 15 20:41:42.589411 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:42.589373    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks" event={"ID":"0dccda17-f48c-46c1-bcba-ea7e6b6bd355","Type":"ContainerStarted","Data":"8a47012563033570edd4630cd7fb06a6eef40d65cea2087379b9d5723f9348e5"}
Sep 15 20:41:42.589885 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:42.589423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks" event={"ID":"0dccda17-f48c-46c1-bcba-ea7e6b6bd355","Type":"ContainerStarted","Data":"4183d3d54e1a8de27027ba652e1fc086895e65e8925b5148605a8f1bad8f145f"}
Sep 15 20:41:42.606614 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:42.606564    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cdlks" podStartSLOduration=4.04592348 podStartE2EDuration="5.606549031s" totalImagesPullingTime="1.560625551s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:42.605796313 +0000 UTC m=+293.392920350" watchObservedRunningTime="2026-09-15 20:41:42.606549031 +0000 UTC m=+293.393673066"
Sep 15 20:41:44.349266 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.349227    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5fdcq"]
Sep 15 20:41:44.352563 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.352546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.355378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.355316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 20:41:44.355508 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.355396    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qq8fj\""
Sep 15 20:41:44.355565 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.355553    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 20:41:44.355675 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.355652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 20:41:44.440698 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440661    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-sys\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.440894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440713    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-wtmp\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.440894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-tls\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.440894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.440894 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-textfile\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.441082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-root\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.441082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440961    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4kxz\" (UniqueName: \"kubernetes.io/projected/a649fb8f-5155-4a38-9979-098122c8990b-kube-api-access-w4kxz\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.441082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.440984    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.441082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.441002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-metrics-client-ca\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542239 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4kxz\" (UniqueName: \"kubernetes.io/projected/a649fb8f-5155-4a38-9979-098122c8990b-kube-api-access-w4kxz\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-metrics-client-ca\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542530 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-sys\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542530 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-wtmp\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542530 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542523    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-tls\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-sys\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-textfile\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-root\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542849 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-wtmp\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542849 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542725    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a649fb8f-5155-4a38-9979-098122c8990b-root\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542962 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-metrics-client-ca\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.542962 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.542928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-textfile\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.543084 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.543049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.544900 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.544877    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-tls\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.545058 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.545041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a649fb8f-5155-4a38-9979-098122c8990b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.549482 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.549454    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4kxz\" (UniqueName: \"kubernetes.io/projected/a649fb8f-5155-4a38-9979-098122c8990b-kube-api-access-w4kxz\") pod \"node-exporter-5fdcq\" (UID: \"a649fb8f-5155-4a38-9979-098122c8990b\") " pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.661717 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:44.661680    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5fdcq"
Sep 15 20:41:44.662903 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:41:44.662874    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-cflrg" podUID="2dc71d14-d23a-437d-ba3d-1edf2ffb7dde"
Sep 15 20:41:44.671664 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:44.671637    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda649fb8f_5155_4a38_9979_098122c8990b.slice/crio-c7bbd37b01040872627937c908cac009cb9b42f289cb2e90f513407817a04c42 WatchSource:0}: Error finding container c7bbd37b01040872627937c908cac009cb9b42f289cb2e90f513407817a04c42: Status 404 returned error can't find the container with id c7bbd37b01040872627937c908cac009cb9b42f289cb2e90f513407817a04c42
Sep 15 20:41:45.602221 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:45.602186    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fdcq" event={"ID":"a649fb8f-5155-4a38-9979-098122c8990b","Type":"ContainerStarted","Data":"c7bbd37b01040872627937c908cac009cb9b42f289cb2e90f513407817a04c42"}
Sep 15 20:41:46.610286 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:46.610248    2579 generic.go:356] "Generic (PLEG): container finished" podID="a649fb8f-5155-4a38-9979-098122c8990b" containerID="92a3a1c6441e368593074228be8259992b03159267c11a6cb8ce7806abb3e10e" exitCode=0
Sep 15 20:41:46.610778 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:46.610333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fdcq" event={"ID":"a649fb8f-5155-4a38-9979-098122c8990b","Type":"ContainerDied","Data":"92a3a1c6441e368593074228be8259992b03159267c11a6cb8ce7806abb3e10e"}
Sep 15 20:41:47.616783 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:47.616750    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fdcq" event={"ID":"a649fb8f-5155-4a38-9979-098122c8990b","Type":"ContainerStarted","Data":"0d84d3bb617002f1bbbc23f7cdfa40fa21e05880506f893e171d58b6f7e13e40"}
Sep 15 20:41:47.617174 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:47.616800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5fdcq" event={"ID":"a649fb8f-5155-4a38-9979-098122c8990b","Type":"ContainerStarted","Data":"7dfddd4fa2899b0e4a8d3a0ef7779763589791ae7ee42b730b577fdba216b21c"}
Sep 15 20:41:47.636925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:47.636871    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5fdcq" podStartSLOduration=2.361979874 podStartE2EDuration="3.636853844s" totalImagesPullingTime="1.206756019s" totalInitContainerRuntime="68.117951ms" isStatefulPod=true podCreationTimestamp="2026-09-15 20:41:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:41:47.635305546 +0000 UTC m=+298.422429588" watchObservedRunningTime="2026-09-15 20:41:47.636853844 +0000 UTC m=+298.423977873"
Sep 15 20:41:48.576278 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:48.576249    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-26p5b"
Sep 15 20:41:49.201701 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:49.201656    2579 patch_prober.go:28] interesting pod/image-registry-6b69dc5c9c-2dkxx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 20:41:49.202119 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:49.201717    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx" podUID="2304952d-987b-4c91-923d-8d3a1f55fcbd" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 20:41:49.575879 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:49.575812    2579 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 20:41:51.548560 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:51.548531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b69dc5c9c-2dkxx"
Sep 15 20:41:51.947473 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:51.947436    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" podUID="acf24153-a5fc-4b79-9815-ef87af90518b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 20:41:55.188378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.188341    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-klgxq"]
Sep 15 20:41:55.191492 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.191475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:41:55.194176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.194155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 20:41:55.194291 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.194241    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h6gxg\""
Sep 15 20:41:55.194338 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.194321    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 20:41:55.201224 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.201203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-klgxq"]
Sep 15 20:41:55.330616 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.330580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2frq\" (UniqueName: \"kubernetes.io/projected/8de405a3-c83e-49ed-91bd-7267bdc52c11-kube-api-access-n2frq\") pod \"downloads-76f99584f4-klgxq\" (UID: \"8de405a3-c83e-49ed-91bd-7267bdc52c11\") " pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:41:55.431804 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.431766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2frq\" (UniqueName: \"kubernetes.io/projected/8de405a3-c83e-49ed-91bd-7267bdc52c11-kube-api-access-n2frq\") pod \"downloads-76f99584f4-klgxq\" (UID: \"8de405a3-c83e-49ed-91bd-7267bdc52c11\") " pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:41:55.450294 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.450222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2frq\" (UniqueName: \"kubernetes.io/projected/8de405a3-c83e-49ed-91bd-7267bdc52c11-kube-api-access-n2frq\") pod \"downloads-76f99584f4-klgxq\" (UID: \"8de405a3-c83e-49ed-91bd-7267bdc52c11\") " pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:41:55.500412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.500382    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:41:55.662703 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.662670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:41:55.785612 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.785519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-klgxq"]
Sep 15 20:41:55.790651 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:41:55.790616    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de405a3_c83e_49ed_91bd_7267bdc52c11.slice/crio-8623d1d164b10bdaff92999220f4b6fca0a29cf39bd18718cdefcd2a69f2275c WatchSource:0}: Error finding container 8623d1d164b10bdaff92999220f4b6fca0a29cf39bd18718cdefcd2a69f2275c: Status 404 returned error can't find the container with id 8623d1d164b10bdaff92999220f4b6fca0a29cf39bd18718cdefcd2a69f2275c
Sep 15 20:41:55.793746 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:55.792964    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 20:41:56.647978 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:41:56.647945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-klgxq" event={"ID":"8de405a3-c83e-49ed-91bd-7267bdc52c11","Type":"ContainerStarted","Data":"8623d1d164b10bdaff92999220f4b6fca0a29cf39bd18718cdefcd2a69f2275c"}
Sep 15 20:42:01.306319 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.306284    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:01.309962 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.309942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.312490 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.312464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 20:42:01.313988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.313925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 20:42:01.313988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.313933    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 20:42:01.313988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.313960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sjxrh\""
Sep 15 20:42:01.313988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.313964    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 20:42:01.314363 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.313925    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 20:42:01.320145 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.320123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:01.382744 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382705    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.382914 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382756    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.382914 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.382914 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.383077 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gx72\" (UniqueName: \"kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.383077 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.382953    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483586 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483754 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gx72\" (UniqueName: \"kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483754 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483854 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483911 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.483973 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.483913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.484751 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.484585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.484751 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.484603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.484751 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.484650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.486603 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.486581    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.486715 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.486699    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.491682 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.491658    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gx72\" (UniqueName: \"kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72\") pod \"console-8499bfc5fd-9rmf6\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") " pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.622285 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.622250    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:01.897237 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.897156    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:01.900780 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:42:01.900751    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6728460b_a6e6_4121_8c75_3b5b85789de8.slice/crio-46957e7ce1032008763ba5395254a3a4f94f78245e0cefce6aff58267556613e WatchSource:0}: Error finding container 46957e7ce1032008763ba5395254a3a4f94f78245e0cefce6aff58267556613e: Status 404 returned error can't find the container with id 46957e7ce1032008763ba5395254a3a4f94f78245e0cefce6aff58267556613e
Sep 15 20:42:01.947294 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:01.947253    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" podUID="acf24153-a5fc-4b79-9815-ef87af90518b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 20:42:02.674355 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:02.674262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499bfc5fd-9rmf6" event={"ID":"6728460b-a6e6-4121-8c75-3b5b85789de8","Type":"ContainerStarted","Data":"46957e7ce1032008763ba5395254a3a4f94f78245e0cefce6aff58267556613e"}
Sep 15 20:42:05.688969 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:05.688932    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499bfc5fd-9rmf6" event={"ID":"6728460b-a6e6-4121-8c75-3b5b85789de8","Type":"ContainerStarted","Data":"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"}
Sep 15 20:42:05.710736 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:05.710679    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8499bfc5fd-9rmf6" podStartSLOduration=1.359301741 podStartE2EDuration="4.710656007s" totalImagesPullingTime="3.351354266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:42:05.709674586 +0000 UTC m=+316.496798622" watchObservedRunningTime="2026-09-15 20:42:05.710656007 +0000 UTC m=+316.497780030"
Sep 15 20:42:11.623231 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.623195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:11.623711 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.623357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:11.629097 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.629075    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:11.714053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.714024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:11.947629 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.947553    2579 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" podUID="acf24153-a5fc-4b79-9815-ef87af90518b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 15 20:42:11.947773 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.947633    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7"
Sep 15 20:42:11.948177 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.948132    2579 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"88aac820a52c0aa1ad80c3c5144506d89c6331b8607c9f3653253357cef3ce6f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 15 20:42:11.948266 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:11.948204    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" podUID="acf24153-a5fc-4b79-9815-ef87af90518b" containerName="service-proxy" containerID="cri-o://88aac820a52c0aa1ad80c3c5144506d89c6331b8607c9f3653253357cef3ce6f" gracePeriod=30
Sep 15 20:42:14.724603 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.724566    2579 generic.go:356] "Generic (PLEG): container finished" podID="acf24153-a5fc-4b79-9815-ef87af90518b" containerID="88aac820a52c0aa1ad80c3c5144506d89c6331b8607c9f3653253357cef3ce6f" exitCode=2
Sep 15 20:42:14.725114 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.724652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerDied","Data":"88aac820a52c0aa1ad80c3c5144506d89c6331b8607c9f3653253357cef3ce6f"}
Sep 15 20:42:14.725114 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.724706    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d4fdd86f6-pzhw7" event={"ID":"acf24153-a5fc-4b79-9815-ef87af90518b","Type":"ContainerStarted","Data":"29bf9c5cf25a3aac3ade9c3af1aab2c04fc4b76323d07f63a803eec106e85257"}
Sep 15 20:42:14.726528 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.726503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-klgxq" event={"ID":"8de405a3-c83e-49ed-91bd-7267bdc52c11","Type":"ContainerStarted","Data":"7c7c4ad8662d83fffff091d53a2e7831e5915104a4d8b26119bfeb56c2374cc8"}
Sep 15 20:42:14.726678 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.726660    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:42:14.745068 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.745033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-klgxq"
Sep 15 20:42:14.764074 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:14.763991    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-klgxq" podStartSLOduration=1.306232889 podStartE2EDuration="19.763971336s" totalImagesPullingTime="18.457738447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:42:14.763467572 +0000 UTC m=+325.550591602" watchObservedRunningTime="2026-09-15 20:42:14.763971336 +0000 UTC m=+325.551095366"
Sep 15 20:42:21.493923 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:21.493887    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:46.519936 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.519832    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8499bfc5fd-9rmf6" podUID="6728460b-a6e6-4121-8c75-3b5b85789de8" containerName="console" containerID="cri-o://a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca" gracePeriod=15
Sep 15 20:42:46.807756 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.807726    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:46.838025 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.837982    2579 generic.go:356] "Generic (PLEG): container finished" podID="6728460b-a6e6-4121-8c75-3b5b85789de8" containerID="a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca" exitCode=0
Sep 15 20:42:46.838185 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.838045    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8499bfc5fd-9rmf6"
Sep 15 20:42:46.838185 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.838047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499bfc5fd-9rmf6" event={"ID":"6728460b-a6e6-4121-8c75-3b5b85789de8","Type":"ContainerDied","Data":"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"}
Sep 15 20:42:46.838185 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.838102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8499bfc5fd-9rmf6" event={"ID":"6728460b-a6e6-4121-8c75-3b5b85789de8","Type":"ContainerDied","Data":"46957e7ce1032008763ba5395254a3a4f94f78245e0cefce6aff58267556613e"}
Sep 15 20:42:46.838185 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.838122    2579 scope.go:117] "RemoveContainer" containerID="a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"
Sep 15 20:42:46.841196 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841175    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.841309 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841205    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.841309 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gx72\" (UniqueName: \"kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.841309 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.841473 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841309    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.841473 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841364    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert\") pod \"6728460b-a6e6-4121-8c75-3b5b85789de8\" (UID: \"6728460b-a6e6-4121-8c75-3b5b85789de8\") "
Sep 15 20:42:46.842036 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config" (OuterVolumeSpecName: "console-config") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:42:46.842036 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:42:46.842036 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.841915    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca" (OuterVolumeSpecName: "service-ca") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:42:46.844171 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.844148    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:42:46.844262 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.844170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:42:46.844262 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.844191    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72" (OuterVolumeSpecName: "kube-api-access-6gx72") pod "6728460b-a6e6-4121-8c75-3b5b85789de8" (UID: "6728460b-a6e6-4121-8c75-3b5b85789de8"). InnerVolumeSpecName "kube-api-access-6gx72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:42:46.847084 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.847068    2579 scope.go:117] "RemoveContainer" containerID="a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"
Sep 15 20:42:46.847378 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:42:46.847351    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca\": container with ID starting with a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca not found: ID does not exist" containerID="a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"
Sep 15 20:42:46.847444 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.847386    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca"} err="failed to get container status \"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca\": rpc error: code = NotFound desc = could not find container \"a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca\": container with ID starting with a1fcf82463c3353cc1037749063f69e00752fd3b6e7786d8fef62fa9d77e80ca not found: ID does not exist"
Sep 15 20:42:46.942464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942429    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-oauth-config\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:46.942464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942458    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:46.942464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942469    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-oauth-serving-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:46.942464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942477    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6728460b-a6e6-4121-8c75-3b5b85789de8-console-serving-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:46.942707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942485    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6728460b-a6e6-4121-8c75-3b5b85789de8-console-config\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:46.942707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:46.942493    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gx72\" (UniqueName: \"kubernetes.io/projected/6728460b-a6e6-4121-8c75-3b5b85789de8-kube-api-access-6gx72\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:42:47.158391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:47.158368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:47.161923 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:47.161904    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8499bfc5fd-9rmf6"]
Sep 15 20:42:47.666173 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:42:47.666145    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6728460b-a6e6-4121-8c75-3b5b85789de8" path="/var/lib/kubelet/pods/6728460b-a6e6-4121-8c75-3b5b85789de8/volumes"
Sep 15 20:43:03.469240 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:03.469201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:43:03.471587 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:03.471566    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dc71d14-d23a-437d-ba3d-1edf2ffb7dde-metrics-certs\") pod \"network-metrics-daemon-cflrg\" (UID: \"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde\") " pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:43:03.765925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:03.765832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nxplx\""
Sep 15 20:43:03.774230 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:03.774199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cflrg"
Sep 15 20:43:04.048608 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:04.048527    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cflrg"]
Sep 15 20:43:04.052542 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:43:04.052516    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc71d14_d23a_437d_ba3d_1edf2ffb7dde.slice/crio-c1f4b3bffd987759e491661aa8372a3dbc0422539770378e195d7a8a23647974 WatchSource:0}: Error finding container c1f4b3bffd987759e491661aa8372a3dbc0422539770378e195d7a8a23647974: Status 404 returned error can't find the container with id c1f4b3bffd987759e491661aa8372a3dbc0422539770378e195d7a8a23647974
Sep 15 20:43:04.898025 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:04.897980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cflrg" event={"ID":"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde","Type":"ContainerStarted","Data":"c1f4b3bffd987759e491661aa8372a3dbc0422539770378e195d7a8a23647974"}
Sep 15 20:43:05.903943 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:05.903915    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cflrg" event={"ID":"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde","Type":"ContainerStarted","Data":"7454edf93ae6e4d87c38612247256d1601de30f267e3f4e672bfa4c21796e155"}
Sep 15 20:43:05.904359 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:05.903972    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cflrg" event={"ID":"2dc71d14-d23a-437d-ba3d-1edf2ffb7dde","Type":"ContainerStarted","Data":"6e0d156ec9bed17686454c72a1415b5b4264d419e2fc5745ff850aa0654af68a"}
Sep 15 20:43:05.920440 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:43:05.920390    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cflrg" podStartSLOduration=375.280901538 podStartE2EDuration="6m16.920375841s" totalImagesPullingTime="1.639474303s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:43:05.91889076 +0000 UTC m=+376.706014790" watchObservedRunningTime="2026-09-15 20:43:05.920375841 +0000 UTC m=+376.707499870"
Sep 15 20:44:29.667566 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.667532    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:44:29.668038 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.667759    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6728460b-a6e6-4121-8c75-3b5b85789de8" containerName="console"
Sep 15 20:44:29.668038 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.667770    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6728460b-a6e6-4121-8c75-3b5b85789de8" containerName="console"
Sep 15 20:44:29.668038 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.667817    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6728460b-a6e6-4121-8c75-3b5b85789de8" containerName="console"
Sep 15 20:44:29.669739 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.669723    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.673194 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 20:44:29.673318 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 20:44:29.673438 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-sjxrh\""
Sep 15 20:44:29.673540 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673420    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 20:44:29.673540 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673472    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 20:44:29.673540 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.673518    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 20:44:29.679035 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.678998    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 20:44:29.681491 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.681472    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:44:29.734382 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5757d\" (UniqueName: \"kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734382 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734382 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734394    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734632 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734632 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734632 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.734632 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.734490    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835672 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5757d\" (UniqueName: \"kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835672 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.835910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.835824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.836695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.836627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.836695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.836650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.836695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.836650    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.836945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.836769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.838353 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.838330    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.838449 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.838428    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.843838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.843820    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5757d\" (UniqueName: \"kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d\") pod \"console-788f8857d7-q6kt9\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") " pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:29.978920 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:29.978827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:30.250399 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:30.250375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:44:30.254580 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:44:30.254555    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764a96d5_8700_4577_8a05_3ac39bf2f03e.slice/crio-fb682b2c5f690bc9d364a39dc0d99701416bc4a0eaf0aacb8b7b4e7ebcbeb789 WatchSource:0}: Error finding container fb682b2c5f690bc9d364a39dc0d99701416bc4a0eaf0aacb8b7b4e7ebcbeb789: Status 404 returned error can't find the container with id fb682b2c5f690bc9d364a39dc0d99701416bc4a0eaf0aacb8b7b4e7ebcbeb789
Sep 15 20:44:31.194706 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:31.194662    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788f8857d7-q6kt9" event={"ID":"764a96d5-8700-4577-8a05-3ac39bf2f03e","Type":"ContainerStarted","Data":"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"}
Sep 15 20:44:31.195217 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:31.194720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788f8857d7-q6kt9" event={"ID":"764a96d5-8700-4577-8a05-3ac39bf2f03e","Type":"ContainerStarted","Data":"fb682b2c5f690bc9d364a39dc0d99701416bc4a0eaf0aacb8b7b4e7ebcbeb789"}
Sep 15 20:44:31.215121 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:31.211217    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-788f8857d7-q6kt9" podStartSLOduration=2.21119701 podStartE2EDuration="2.21119701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:44:31.210512282 +0000 UTC m=+461.997636323" watchObservedRunningTime="2026-09-15 20:44:31.21119701 +0000 UTC m=+461.998321041"
Sep 15 20:44:39.979307 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:39.979262    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:39.979698 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:39.979336    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:39.983848 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:39.983827    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:44:40.229180 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:44:40.229153    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:45:49.972110 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.972076    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"]
Sep 15 20:45:49.973966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.973943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:49.976925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.976901    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4jvm7\""
Sep 15 20:45:49.976925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.976903    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 15 20:45:49.977116 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.976938    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:45:49.990623 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:49.990593    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"]
Sep 15 20:45:50.088916 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.088875    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmxxj\" (UniqueName: \"kubernetes.io/projected/86fef466-5762-4443-9b29-0c15d97c8af0-kube-api-access-pmxxj\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.089130 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.088928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86fef466-5762-4443-9b29-0c15d97c8af0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.190148 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.190110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86fef466-5762-4443-9b29-0c15d97c8af0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.190314 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.190172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmxxj\" (UniqueName: \"kubernetes.io/projected/86fef466-5762-4443-9b29-0c15d97c8af0-kube-api-access-pmxxj\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.190484 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.190465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/86fef466-5762-4443-9b29-0c15d97c8af0-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.199547 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.199515    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmxxj\" (UniqueName: \"kubernetes.io/projected/86fef466-5762-4443-9b29-0c15d97c8af0-kube-api-access-pmxxj\") pod \"cert-manager-operator-controller-manager-55c444bf86-smwtv\" (UID: \"86fef466-5762-4443-9b29-0c15d97c8af0\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.287412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.287322    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"
Sep 15 20:45:50.570887 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:50.570808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv"]
Sep 15 20:45:50.573974 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:45:50.573946    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fef466_5762_4443_9b29_0c15d97c8af0.slice/crio-dca48f03e2757b74d97f1582587bc306c9cb4d40740485dbe3e682d88be3c35e WatchSource:0}: Error finding container dca48f03e2757b74d97f1582587bc306c9cb4d40740485dbe3e682d88be3c35e: Status 404 returned error can't find the container with id dca48f03e2757b74d97f1582587bc306c9cb4d40740485dbe3e682d88be3c35e
Sep 15 20:45:51.463158 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:51.463118    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv" event={"ID":"86fef466-5762-4443-9b29-0c15d97c8af0","Type":"ContainerStarted","Data":"dca48f03e2757b74d97f1582587bc306c9cb4d40740485dbe3e682d88be3c35e"}
Sep 15 20:45:53.473372 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:53.473293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv" event={"ID":"86fef466-5762-4443-9b29-0c15d97c8af0","Type":"ContainerStarted","Data":"12e24187a8fec526a3e82e2a6d56494107a5669a03ee20dbf34cd9d671efdd3f"}
Sep 15 20:45:53.495410 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:45:53.495363    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-smwtv" podStartSLOduration=1.876888171 podStartE2EDuration="4.495348144s" totalImagesPullingTime="2.618459973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:45:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:45:53.49327128 +0000 UTC m=+544.280395310" watchObservedRunningTime="2026-09-15 20:45:53.495348144 +0000 UTC m=+544.282472172"
Sep 15 20:46:12.300945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.300912    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"]
Sep 15 20:46:12.303902 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.303883    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.306474 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.306451    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 15 20:46:12.306595 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.306550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-wnbz4\""
Sep 15 20:46:12.307673 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.307656    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 15 20:46:12.311782 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.311764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"]
Sep 15 20:46:12.451846 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.451801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0751e3b5-50da-4d2f-92ef-334587009b21-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.452107 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.451857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxs6l\" (UniqueName: \"kubernetes.io/projected/0751e3b5-50da-4d2f-92ef-334587009b21-kube-api-access-hxs6l\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.553137 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.553061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0751e3b5-50da-4d2f-92ef-334587009b21-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.553137 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.553102    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxs6l\" (UniqueName: \"kubernetes.io/projected/0751e3b5-50da-4d2f-92ef-334587009b21-kube-api-access-hxs6l\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.553477 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.553456    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0751e3b5-50da-4d2f-92ef-334587009b21-tmp\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.564507 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.564479    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxs6l\" (UniqueName: \"kubernetes.io/projected/0751e3b5-50da-4d2f-92ef-334587009b21-kube-api-access-hxs6l\") pod \"openshift-lws-operator-fd8ccff4c-24tww\" (UID: \"0751e3b5-50da-4d2f-92ef-334587009b21\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.613087 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.613039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"
Sep 15 20:46:12.881891 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:12.881852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww"]
Sep 15 20:46:12.884658 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:46:12.884632    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0751e3b5_50da_4d2f_92ef_334587009b21.slice/crio-b9e5d5e8f3be3196efe119c54924ebcc22b2d1224c3a1e8188ac9f42d635046c WatchSource:0}: Error finding container b9e5d5e8f3be3196efe119c54924ebcc22b2d1224c3a1e8188ac9f42d635046c: Status 404 returned error can't find the container with id b9e5d5e8f3be3196efe119c54924ebcc22b2d1224c3a1e8188ac9f42d635046c
Sep 15 20:46:13.543697 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:13.543663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww" event={"ID":"0751e3b5-50da-4d2f-92ef-334587009b21","Type":"ContainerStarted","Data":"b9e5d5e8f3be3196efe119c54924ebcc22b2d1224c3a1e8188ac9f42d635046c"}
Sep 15 20:46:15.553514 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:15.553476    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww" event={"ID":"0751e3b5-50da-4d2f-92ef-334587009b21","Type":"ContainerStarted","Data":"4f697576b5f4f7a33484cd8aead4961cafa98763576813875b43bc83fa844f95"}
Sep 15 20:46:15.569959 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:15.569911    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-24tww" podStartSLOduration=1.074349564 podStartE2EDuration="3.569897205s" totalImagesPullingTime="2.495547641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:15.568717178 +0000 UTC m=+566.355841208" watchObservedRunningTime="2026-09-15 20:46:15.569897205 +0000 UTC m=+566.357021233"
Sep 15 20:46:31.999978 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:31.999946    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"]
Sep 15 20:46:32.004901 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.004879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.005664 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.005642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.005743 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.005682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.005743 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.005699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.005816 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.005772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.005816 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.005792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fln6\" (UniqueName: \"kubernetes.io/projected/046286e0-b330-4533-a2e2-829caf6cf5be-kube-api-access-7fln6\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.007578 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.007558    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fg66b\""
Sep 15 20:46:32.007673 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.007620    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 15 20:46:32.007805 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.007788    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 15 20:46:32.007866 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.007804    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 15 20:46:32.008205 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.008190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 15 20:46:32.020998 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.020978    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"]
Sep 15 20:46:32.106284 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.106284 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.106518 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.106518 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106341    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.106518 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fln6\" (UniqueName: \"kubernetes.io/projected/046286e0-b330-4533-a2e2-829caf6cf5be-kube-api-access-7fln6\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.106773 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.106741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-manifests\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.108752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.108733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/046286e0-b330-4533-a2e2-829caf6cf5be-tmp\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.109027 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.108990    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-webhook-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.109095 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.109025    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/046286e0-b330-4533-a2e2-829caf6cf5be-apiservice-cert\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.115049 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.115001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fln6\" (UniqueName: \"kubernetes.io/projected/046286e0-b330-4533-a2e2-829caf6cf5be-kube-api-access-7fln6\") pod \"opendatahub-operator-controller-manager-687b5f4fc4-9wbv6\" (UID: \"046286e0-b330-4533-a2e2-829caf6cf5be\") " pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.315084 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.314975    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:32.593690 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.593663    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"]
Sep 15 20:46:32.595367 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:46:32.595332    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod046286e0_b330_4533_a2e2_829caf6cf5be.slice/crio-d3de47200b5a6f4af927245e333aaa63662d8a09a0bf1e21acfd14b38b818785 WatchSource:0}: Error finding container d3de47200b5a6f4af927245e333aaa63662d8a09a0bf1e21acfd14b38b818785: Status 404 returned error can't find the container with id d3de47200b5a6f4af927245e333aaa63662d8a09a0bf1e21acfd14b38b818785
Sep 15 20:46:32.618491 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:32.618456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6" event={"ID":"046286e0-b330-4533-a2e2-829caf6cf5be","Type":"ContainerStarted","Data":"d3de47200b5a6f4af927245e333aaa63662d8a09a0bf1e21acfd14b38b818785"}
Sep 15 20:46:36.636654 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:36.636618    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6" event={"ID":"046286e0-b330-4533-a2e2-829caf6cf5be","Type":"ContainerStarted","Data":"0d9404e71b5e5a0d490ba709e51422685d3a17caafe5dccb1fa4d64eaa7e79ea"}
Sep 15 20:46:37.642440 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:37.642405    2579 generic.go:356] "Generic (PLEG): container finished" podID="046286e0-b330-4533-a2e2-829caf6cf5be" containerID="0d9404e71b5e5a0d490ba709e51422685d3a17caafe5dccb1fa4d64eaa7e79ea" exitCode=0
Sep 15 20:46:37.643031 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:37.642461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6" event={"ID":"046286e0-b330-4533-a2e2-829caf6cf5be","Type":"ContainerDied","Data":"0d9404e71b5e5a0d490ba709e51422685d3a17caafe5dccb1fa4d64eaa7e79ea"}
Sep 15 20:46:38.648017 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:38.647974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6" event={"ID":"046286e0-b330-4533-a2e2-829caf6cf5be","Type":"ContainerStarted","Data":"d357379ec74f30cb151ce8871f0bf5067510b9bc275fac93846a9840192aeaf7"}
Sep 15 20:46:38.648383 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:38.648064    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:38.672281 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:38.672238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6" podStartSLOduration=3.100327028 podStartE2EDuration="7.672225034s" totalImagesPullingTime="3.531228979s" totalInitContainerRuntime="1.040669027s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:38.670782096 +0000 UTC m=+589.457906123" watchObservedRunningTime="2026-09-15 20:46:38.672225034 +0000 UTC m=+589.459349062"
Sep 15 20:46:49.654689 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:49.654654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-687b5f4fc4-9wbv6"
Sep 15 20:46:52.442373 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.442335    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"]
Sep 15 20:46:52.445048 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.445031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.448459 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.448437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 15 20:46:52.448590 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.448568    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 15 20:46:52.448851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.448834    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 20:46:52.449467 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.449453    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 20:46:52.458112 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.458088    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"]
Sep 15 20:46:52.549065 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.548997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8zdf\" (UniqueName: \"kubernetes.io/projected/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-kube-api-access-p8zdf\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.549251 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.549076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tls-certs\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.549251 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.549121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tmp\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.650335 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.650284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tmp\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.650495 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.650381    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8zdf\" (UniqueName: \"kubernetes.io/projected/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-kube-api-access-p8zdf\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.650495 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.650405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tls-certs\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.652809 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.652787    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tmp\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.653079 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.653062    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-tls-certs\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.659049 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.659026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8zdf\" (UniqueName: \"kubernetes.io/projected/7fb4dafb-3c2b-4509-91b7-dd823ac52ee4-kube-api-access-p8zdf\") pod \"kube-auth-proxy-65544b76bd-2xvl7\" (UID: \"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4\") " pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:52.754994 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:52.754908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"
Sep 15 20:46:53.025201 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.025103    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7"]
Sep 15 20:46:53.027957 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:46:53.027932    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb4dafb_3c2b_4509_91b7_dd823ac52ee4.slice/crio-6ed9b4555b14669e74fc2458d0fe0f3426d926c6bf70fd471e27b56fcdb499d1 WatchSource:0}: Error finding container 6ed9b4555b14669e74fc2458d0fe0f3426d926c6bf70fd471e27b56fcdb499d1: Status 404 returned error can't find the container with id 6ed9b4555b14669e74fc2458d0fe0f3426d926c6bf70fd471e27b56fcdb499d1
Sep 15 20:46:53.708653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.708619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7" event={"ID":"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4","Type":"ContainerStarted","Data":"6ed9b4555b14669e74fc2458d0fe0f3426d926c6bf70fd471e27b56fcdb499d1"}
Sep 15 20:46:53.720334 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.720303    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:46:53.723482 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.723457    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.726463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.726315    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 15 20:46:53.726463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.726333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 15 20:46:53.726463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.726364    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 15 20:46:53.726463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.726315    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 15 20:46:53.727001 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.726982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-9kbm6\""
Sep 15 20:46:53.737514 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.737492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:46:53.760588 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.760774 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760635    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.760774 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760663    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.760774 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760712    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.761043 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.761043 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.761043 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.760899    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6wg\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861901    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6wg\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.862555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.861985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.863255 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.862807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.865026 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.864981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.865265 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.865221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.865383 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.865290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.867925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.866683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.872479 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.872438    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6wg\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:53.873117 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:53.873075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qplft\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:54.037728 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:54.037635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:54.330810 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:54.330714    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:46:54.333740 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:46:54.333708    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114e998a_bf44_48cf_ae1c_17f04528a5c0.slice/crio-13e43d2db54a840b00d442a0b53501f5c3d924406e8cb0b186798422a42e104e WatchSource:0}: Error finding container 13e43d2db54a840b00d442a0b53501f5c3d924406e8cb0b186798422a42e104e: Status 404 returned error can't find the container with id 13e43d2db54a840b00d442a0b53501f5c3d924406e8cb0b186798422a42e104e
Sep 15 20:46:54.714604 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:54.714542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" event={"ID":"114e998a-bf44-48cf-ae1c-17f04528a5c0","Type":"ContainerStarted","Data":"13e43d2db54a840b00d442a0b53501f5c3d924406e8cb0b186798422a42e104e"}
Sep 15 20:46:56.726094 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:56.726055    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7" event={"ID":"7fb4dafb-3c2b-4509-91b7-dd823ac52ee4","Type":"ContainerStarted","Data":"35a16c67e54d730bb35b0eca2ef921a438740b3d1305b532a112d486269c59a5"}
Sep 15 20:46:56.741717 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:56.741656    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-65544b76bd-2xvl7" podStartSLOduration=1.976186293 podStartE2EDuration="4.741632659s" totalImagesPullingTime="2.765446366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:56.741475025 +0000 UTC m=+607.528599055" watchObservedRunningTime="2026-09-15 20:46:56.741632659 +0000 UTC m=+607.528756690"
Sep 15 20:46:57.529592 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:57.529544    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 20:46:57.529705 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:57.529629    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 20:46:57.731609 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:57.731570    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" event={"ID":"114e998a-bf44-48cf-ae1c-17f04528a5c0","Type":"ContainerStarted","Data":"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"}
Sep 15 20:46:57.731974 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:57.731796    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:46:57.752439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:57.752382    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" podStartSLOduration=1.559550324 podStartE2EDuration="4.752366199s" totalImagesPullingTime="3.192815875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:46:57.750705535 +0000 UTC m=+608.537829565" watchObservedRunningTime="2026-09-15 20:46:57.752366199 +0000 UTC m=+608.539490227"
Sep 15 20:46:58.737595 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:58.737558    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qplft container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 15 20:46:58.737976 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:46:58.737618    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 20:47:01.737036 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:01.736984    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:47:10.682979 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.682941    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"]
Sep 15 20:47:10.690036 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.689996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.698116 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.698091    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"]
Sep 15 20:47:10.804664 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804664 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804666    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkz2\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-kube-api-access-kzkz2\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804790    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.804912 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.804910    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905667 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905667 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905882 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905882 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905791    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905882 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905808    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkz2\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-kube-api-access-kzkz2\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.905882 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.906122 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.905976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.906704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.906676    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.908270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.908249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.908270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.908257    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.908425 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.908360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.908425 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.908375    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.917979 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.917960    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.933880 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.933818    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkz2\" (UniqueName: \"kubernetes.io/projected/0dc80540-cdb8-4e1e-bab3-2b5c5f493332-kube-api-access-kzkz2\") pod \"istiod-openshift-gateway-5677ccb49b-9gzfd\" (UID: \"0dc80540-cdb8-4e1e-bab3-2b5c5f493332\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:10.999448 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:10.999413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:11.283543 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.283473    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"]
Sep 15 20:47:11.286365 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:47:11.286340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc80540_cdb8_4e1e_bab3_2b5c5f493332.slice/crio-dbcd49b07f585c40d4e53c22d68745703c60c93a8f5b8134a54fc9e1d55a39e3 WatchSource:0}: Error finding container dbcd49b07f585c40d4e53c22d68745703c60c93a8f5b8134a54fc9e1d55a39e3: Status 404 returned error can't find the container with id dbcd49b07f585c40d4e53c22d68745703c60c93a8f5b8134a54fc9e1d55a39e3
Sep 15 20:47:11.289611 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.289579    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 20:47:11.289696 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.289640    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 15 20:47:11.790735 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.790699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd" event={"ID":"0dc80540-cdb8-4e1e-bab3-2b5c5f493332","Type":"ContainerStarted","Data":"c52c9cb42138c91ba59950a7631719209bf68c5a886fc37a6db4c5bb428cdc08"}
Sep 15 20:47:11.791207 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.790753    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd" event={"ID":"0dc80540-cdb8-4e1e-bab3-2b5c5f493332","Type":"ContainerStarted","Data":"dbcd49b07f585c40d4e53c22d68745703c60c93a8f5b8134a54fc9e1d55a39e3"}
Sep 15 20:47:11.791207 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.790831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:11.831152 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:11.831079    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd" podStartSLOduration=1.831060177 podStartE2EDuration="1.831060177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:11.83086684 +0000 UTC m=+622.617990870" watchObservedRunningTime="2026-09-15 20:47:11.831060177 +0000 UTC m=+622.618184207"
Sep 15 20:47:12.796648 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:12.796623    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9gzfd"
Sep 15 20:47:12.864091 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:12.864063    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:47:12.864262 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:12.864238    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerName="discovery" containerID="cri-o://cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6" gracePeriod=30
Sep 15 20:47:13.114797 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.114760    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:47:13.225488 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225449    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225488 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c6wg\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225686 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225686 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225662    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225808 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225731    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225808 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225784    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.225926 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.225903    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap\") pod \"114e998a-bf44-48cf-ae1c-17f04528a5c0\" (UID: \"114e998a-bf44-48cf-ae1c-17f04528a5c0\") "
Sep 15 20:47:13.226503 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.226388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:47:13.228168 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228119    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts" (OuterVolumeSpecName: "cacerts") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:47:13.228365 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs" (OuterVolumeSpecName: "local-certs") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 20:47:13.228365 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228343    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:47:13.228529 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228390    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg" (OuterVolumeSpecName: "kube-api-access-7c6wg") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "kube-api-access-7c6wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:47:13.228529 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228444    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token" (OuterVolumeSpecName: "istio-token") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:47:13.228601 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.228551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "114e998a-bf44-48cf-ae1c-17f04528a5c0" (UID: "114e998a-bf44-48cf-ae1c-17f04528a5c0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:47:13.326835 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326797    2579 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/114e998a-bf44-48cf-ae1c-17f04528a5c0-local-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.326835 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326826    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-token\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.326835 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326834    2579 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-kubeconfig\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.326835 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326844    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-dns-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.327112 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326854    2579 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/114e998a-bf44-48cf-ae1c-17f04528a5c0-istio-csr-ca-configmap\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.327112 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326864    2579 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/114e998a-bf44-48cf-ae1c-17f04528a5c0-cacerts\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.327112 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.326872    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c6wg\" (UniqueName: \"kubernetes.io/projected/114e998a-bf44-48cf-ae1c-17f04528a5c0-kube-api-access-7c6wg\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:13.418754 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.418211    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c75569b7c-lg46q"]
Sep 15 20:47:13.418754 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.418652    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerName="discovery"
Sep 15 20:47:13.418754 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.418672    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerName="discovery"
Sep 15 20:47:13.419050 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.418764    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerName="discovery"
Sep 15 20:47:13.421115 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.421090    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.439084 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.439052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c75569b7c-lg46q"]
Sep 15 20:47:13.467330 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.467290    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"]
Sep 15 20:47:13.469508 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.469493    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.472380 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.472357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 15 20:47:13.472521 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.472360    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p9t94\""
Sep 15 20:47:13.480995 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.480969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"]
Sep 15 20:47:13.528359 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-oauth-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n28n5\" (UniqueName: \"kubernetes.io/projected/b38566c6-7cef-40ee-9960-d5aef815ab3b-kube-api-access-n28n5\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-trusted-ca-bundle\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-service-ca\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-oauth-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mglh9\" (UniqueName: \"kubernetes.io/projected/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-kube-api-access-mglh9\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.528921 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.528606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629759 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629718    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-oauth-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n28n5\" (UniqueName: \"kubernetes.io/projected/b38566c6-7cef-40ee-9960-d5aef815ab3b-kube-api-access-n28n5\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-trusted-ca-bundle\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-service-ca\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-oauth-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.629966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mglh9\" (UniqueName: \"kubernetes.io/projected/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-kube-api-access-mglh9\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.630342 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.629987    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.630721 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.630687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.630832 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:47:13.630801    2579 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 15 20:47:13.630891 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:47:13.630864    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert podName:b38566c6-7cef-40ee-9960-d5aef815ab3b nodeName:}" failed. No retries permitted until 2026-09-15 20:47:14.130842405 +0000 UTC m=+624.917966426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert") pod "kserve-controller-manager-d6f4b65b8-w8bf5" (UID: "b38566c6-7cef-40ee-9960-d5aef815ab3b") : secret "kserve-webhook-server-cert" not found
Sep 15 20:47:13.631919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.631881    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-oauth-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.632318 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.632288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-trusted-ca-bundle\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.632408 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.632288    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-service-ca\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.632671 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.632646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-oauth-config\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.632940 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.632919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-console-serving-cert\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.643260 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.643236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n28n5\" (UniqueName: \"kubernetes.io/projected/b38566c6-7cef-40ee-9960-d5aef815ab3b-kube-api-access-n28n5\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:13.643868 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.643846    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mglh9\" (UniqueName: \"kubernetes.io/projected/7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea-kube-api-access-mglh9\") pod \"console-6c75569b7c-lg46q\" (UID: \"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea\") " pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.732549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.732519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:13.802721 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.802690    2579 generic.go:356] "Generic (PLEG): container finished" podID="114e998a-bf44-48cf-ae1c-17f04528a5c0" containerID="cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6" exitCode=0
Sep 15 20:47:13.803174 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.802760    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"
Sep 15 20:47:13.803174 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.802881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" event={"ID":"114e998a-bf44-48cf-ae1c-17f04528a5c0","Type":"ContainerDied","Data":"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"}
Sep 15 20:47:13.803174 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.802950    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft" event={"ID":"114e998a-bf44-48cf-ae1c-17f04528a5c0","Type":"ContainerDied","Data":"13e43d2db54a840b00d442a0b53501f5c3d924406e8cb0b186798422a42e104e"}
Sep 15 20:47:13.803174 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.802977    2579 scope.go:117] "RemoveContainer" containerID="cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"
Sep 15 20:47:13.811928 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.811906    2579 scope.go:117] "RemoveContainer" containerID="cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"
Sep 15 20:47:13.812278 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:47:13.812252    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6\": container with ID starting with cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6 not found: ID does not exist" containerID="cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"
Sep 15 20:47:13.812367 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.812291    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6"} err="failed to get container status \"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6\": rpc error: code = NotFound desc = could not find container \"cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6\": container with ID starting with cfdefbfcdcd02b5958f8f74a10f37433c8fc1777af42c8d15039ef44f5f9c7a6 not found: ID does not exist"
Sep 15 20:47:13.823884 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.823861    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:47:13.828547 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:13.828524    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qplft"]
Sep 15 20:47:14.002908 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.002687    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c75569b7c-lg46q"]
Sep 15 20:47:14.005304 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:47:14.005282    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1f8f2e_e8e2_4f47_9fcf_35bfa7b85dea.slice/crio-ce61a5cd2f38511b8f11e22247aa7d51615fcbb744f8e2c21583fbda8a51f4fe WatchSource:0}: Error finding container ce61a5cd2f38511b8f11e22247aa7d51615fcbb744f8e2c21583fbda8a51f4fe: Status 404 returned error can't find the container with id ce61a5cd2f38511b8f11e22247aa7d51615fcbb744f8e2c21583fbda8a51f4fe
Sep 15 20:47:14.134548 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.134518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:14.137071 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.137049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b38566c6-7cef-40ee-9960-d5aef815ab3b-cert\") pod \"kserve-controller-manager-d6f4b65b8-w8bf5\" (UID: \"b38566c6-7cef-40ee-9960-d5aef815ab3b\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:14.382143 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.382109    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:14.656945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.656823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"]
Sep 15 20:47:14.659389 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:47:14.659363    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38566c6_7cef_40ee_9960_d5aef815ab3b.slice/crio-ed78a3d111e74d45e4afe928047e1f9aae6a0d9b3425d414861d421a38ba7a50 WatchSource:0}: Error finding container ed78a3d111e74d45e4afe928047e1f9aae6a0d9b3425d414861d421a38ba7a50: Status 404 returned error can't find the container with id ed78a3d111e74d45e4afe928047e1f9aae6a0d9b3425d414861d421a38ba7a50
Sep 15 20:47:14.660803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.660787    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 20:47:14.808287 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.808253    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c75569b7c-lg46q" event={"ID":"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea","Type":"ContainerStarted","Data":"0c9376c7ec3f73a67045be90e4ea20b6d557e8f31b04dc28a38150f6b5d6007f"}
Sep 15 20:47:14.808713 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.808308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c75569b7c-lg46q" event={"ID":"7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea","Type":"ContainerStarted","Data":"ce61a5cd2f38511b8f11e22247aa7d51615fcbb744f8e2c21583fbda8a51f4fe"}
Sep 15 20:47:14.810284 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.810255    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5" event={"ID":"b38566c6-7cef-40ee-9960-d5aef815ab3b","Type":"ContainerStarted","Data":"ed78a3d111e74d45e4afe928047e1f9aae6a0d9b3425d414861d421a38ba7a50"}
Sep 15 20:47:14.842785 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:14.842741    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c75569b7c-lg46q" podStartSLOduration=1.842726335 podStartE2EDuration="1.842726335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:14.842704781 +0000 UTC m=+625.629828810" watchObservedRunningTime="2026-09-15 20:47:14.842726335 +0000 UTC m=+625.629850363"
Sep 15 20:47:15.668949 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:15.668913    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114e998a-bf44-48cf-ae1c-17f04528a5c0" path="/var/lib/kubelet/pods/114e998a-bf44-48cf-ae1c-17f04528a5c0/volumes"
Sep 15 20:47:17.825200 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:17.825161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5" event={"ID":"b38566c6-7cef-40ee-9960-d5aef815ab3b","Type":"ContainerStarted","Data":"0ecfbaa264068e55faedb5fa981ebb79c1dc478fc043e4880cee52c88ebc0884"}
Sep 15 20:47:17.825599 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:17.825306    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:17.841633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:17.841589    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5" podStartSLOduration=2.47184864 podStartE2EDuration="4.84157493s" totalImagesPullingTime="2.36972629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:47:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:47:17.84058538 +0000 UTC m=+628.627709410" watchObservedRunningTime="2026-09-15 20:47:17.84157493 +0000 UTC m=+628.628698959"
Sep 15 20:47:23.733568 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:23.733530    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:23.733953 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:23.733594    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:23.738372 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:23.738348    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:23.853367 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:23.853339    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c75569b7c-lg46q"
Sep 15 20:47:23.933431 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:23.933404    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:47:48.833824 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:48.833791    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-w8bf5"
Sep 15 20:47:48.952267 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:48.952228    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-788f8857d7-q6kt9" podUID="764a96d5-8700-4577-8a05-3ac39bf2f03e" containerName="console" containerID="cri-o://cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32" gracePeriod=15
Sep 15 20:47:49.191378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.191354    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:47:49.221557 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221528    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5757d\" (UniqueName: \"kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221709 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221709 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221634    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221709 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221666    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221709 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.221919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.221792    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config\") pod \"764a96d5-8700-4577-8a05-3ac39bf2f03e\" (UID: \"764a96d5-8700-4577-8a05-3ac39bf2f03e\") "
Sep 15 20:47:49.222159 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.222056    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca" (OuterVolumeSpecName: "service-ca") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:47:49.222159 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.222136    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config" (OuterVolumeSpecName: "console-config") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:47:49.222455 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.222267    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:47:49.222710 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.222686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:47:49.223844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.223821    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d" (OuterVolumeSpecName: "kube-api-access-5757d") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "kube-api-access-5757d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:47:49.223980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.223962    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:47:49.224082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.224059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "764a96d5-8700-4577-8a05-3ac39bf2f03e" (UID: "764a96d5-8700-4577-8a05-3ac39bf2f03e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:47:49.323242 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323210    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-oauth-serving-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323242 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323234    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-serving-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323242 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323245    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-config\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323253    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-trusted-ca-bundle\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323261    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/764a96d5-8700-4577-8a05-3ac39bf2f03e-console-oauth-config\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323279    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5757d\" (UniqueName: \"kubernetes.io/projected/764a96d5-8700-4577-8a05-3ac39bf2f03e-kube-api-access-5757d\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.323464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.323288    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/764a96d5-8700-4577-8a05-3ac39bf2f03e-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:47:49.960160 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.960128    2579 generic.go:356] "Generic (PLEG): container finished" podID="764a96d5-8700-4577-8a05-3ac39bf2f03e" containerID="cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32" exitCode=0
Sep 15 20:47:49.960549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.960177    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-788f8857d7-q6kt9"
Sep 15 20:47:49.960549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.960205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788f8857d7-q6kt9" event={"ID":"764a96d5-8700-4577-8a05-3ac39bf2f03e","Type":"ContainerDied","Data":"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"}
Sep 15 20:47:49.960549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.960244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-788f8857d7-q6kt9" event={"ID":"764a96d5-8700-4577-8a05-3ac39bf2f03e","Type":"ContainerDied","Data":"fb682b2c5f690bc9d364a39dc0d99701416bc4a0eaf0aacb8b7b4e7ebcbeb789"}
Sep 15 20:47:49.960549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.960263    2579 scope.go:117] "RemoveContainer" containerID="cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"
Sep 15 20:47:49.968391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.968372    2579 scope.go:117] "RemoveContainer" containerID="cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"
Sep 15 20:47:49.968659 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:47:49.968639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32\": container with ID starting with cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32 not found: ID does not exist" containerID="cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"
Sep 15 20:47:49.968706 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.968672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32"} err="failed to get container status \"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32\": rpc error: code = NotFound desc = could not find container \"cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32\": container with ID starting with cc57b85473bcdfad2252017c3be510935a808c52c2d0bba89a75c853e9499b32 not found: ID does not exist"
Sep 15 20:47:49.976427 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.976394    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:47:49.981045 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:49.980388    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-788f8857d7-q6kt9"]
Sep 15 20:47:51.667621 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:47:51.667590    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="764a96d5-8700-4577-8a05-3ac39bf2f03e" path="/var/lib/kubelet/pods/764a96d5-8700-4577-8a05-3ac39bf2f03e/volumes"
Sep 15 20:48:20.819304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.819269    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-d6w9v"]
Sep 15 20:48:20.819753 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.819542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="764a96d5-8700-4577-8a05-3ac39bf2f03e" containerName="console"
Sep 15 20:48:20.819753 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.819552    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="764a96d5-8700-4577-8a05-3ac39bf2f03e" containerName="console"
Sep 15 20:48:20.819753 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.819605    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="764a96d5-8700-4577-8a05-3ac39bf2f03e" containerName="console"
Sep 15 20:48:20.822404 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.822389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:20.825306 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.825288    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 15 20:48:20.825428 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.825344    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ctdm9\""
Sep 15 20:48:20.825617 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.825604    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 15 20:48:20.836051 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.836000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-d6w9v"]
Sep 15 20:48:20.973313 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:20.973279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhxgl\" (UniqueName: \"kubernetes.io/projected/5289a7c6-c9a0-4230-bb82-84707da7bb41-kube-api-access-jhxgl\") pod \"authorino-operator-76b478bd78-d6w9v\" (UID: \"5289a7c6-c9a0-4230-bb82-84707da7bb41\") " pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:21.074342 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:21.074260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhxgl\" (UniqueName: \"kubernetes.io/projected/5289a7c6-c9a0-4230-bb82-84707da7bb41-kube-api-access-jhxgl\") pod \"authorino-operator-76b478bd78-d6w9v\" (UID: \"5289a7c6-c9a0-4230-bb82-84707da7bb41\") " pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:21.086133 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:21.086108    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhxgl\" (UniqueName: \"kubernetes.io/projected/5289a7c6-c9a0-4230-bb82-84707da7bb41-kube-api-access-jhxgl\") pod \"authorino-operator-76b478bd78-d6w9v\" (UID: \"5289a7c6-c9a0-4230-bb82-84707da7bb41\") " pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:21.132049 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:21.132001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:21.407388 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:21.407365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-d6w9v"]
Sep 15 20:48:21.409657 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:48:21.409629    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5289a7c6_c9a0_4230_bb82_84707da7bb41.slice/crio-ec1fcc8a3f74b2c1d5c1d3d3626c6c94bbf58bcf266ebc5ffb37f39db2391f4f WatchSource:0}: Error finding container ec1fcc8a3f74b2c1d5c1d3d3626c6c94bbf58bcf266ebc5ffb37f39db2391f4f: Status 404 returned error can't find the container with id ec1fcc8a3f74b2c1d5c1d3d3626c6c94bbf58bcf266ebc5ffb37f39db2391f4f
Sep 15 20:48:22.091913 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:22.091879    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v" event={"ID":"5289a7c6-c9a0-4230-bb82-84707da7bb41","Type":"ContainerStarted","Data":"ec1fcc8a3f74b2c1d5c1d3d3626c6c94bbf58bcf266ebc5ffb37f39db2391f4f"}
Sep 15 20:48:24.101599 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:24.101514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v" event={"ID":"5289a7c6-c9a0-4230-bb82-84707da7bb41","Type":"ContainerStarted","Data":"801b5cc904ecefead1c2ac1ada7d0dd6c03885a2da1f280585bbd97f06a03531"}
Sep 15 20:48:24.101945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:24.101648    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:24.119322 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:24.119271    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v" podStartSLOduration=1.705695389 podStartE2EDuration="4.119254687s" totalImagesPullingTime="2.413559298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:24.117957539 +0000 UTC m=+694.905081570" watchObservedRunningTime="2026-09-15 20:48:24.119254687 +0000 UTC m=+694.906378693"
Sep 15 20:48:29.255963 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.255927    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"]
Sep 15 20:48:29.259368 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.259347    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:29.262151 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.262129    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 15 20:48:29.262250 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.262182    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bhslt\""
Sep 15 20:48:29.271161 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.271141    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"]
Sep 15 20:48:29.347849 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.347808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64j7v\" (UniqueName: \"kubernetes.io/projected/3c4236ea-8bca-486c-877d-fe01d9f2d9be-kube-api-access-64j7v\") pod \"dns-operator-controller-manager-7b84cd7c9c-bc9ht\" (UID: \"3c4236ea-8bca-486c-877d-fe01d9f2d9be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:29.449181 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.449149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64j7v\" (UniqueName: \"kubernetes.io/projected/3c4236ea-8bca-486c-877d-fe01d9f2d9be-kube-api-access-64j7v\") pod \"dns-operator-controller-manager-7b84cd7c9c-bc9ht\" (UID: \"3c4236ea-8bca-486c-877d-fe01d9f2d9be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:29.460074 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.460040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64j7v\" (UniqueName: \"kubernetes.io/projected/3c4236ea-8bca-486c-877d-fe01d9f2d9be-kube-api-access-64j7v\") pod \"dns-operator-controller-manager-7b84cd7c9c-bc9ht\" (UID: \"3c4236ea-8bca-486c-877d-fe01d9f2d9be\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:29.569549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.569470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:29.842162 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:29.842140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"]
Sep 15 20:48:29.844709 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:48:29.844677    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4236ea_8bca_486c_877d_fe01d9f2d9be.slice/crio-a9c67262d18efd93c5bae75e748e0ecaa340d831f79df1fefafbf7fd0f08e21d WatchSource:0}: Error finding container a9c67262d18efd93c5bae75e748e0ecaa340d831f79df1fefafbf7fd0f08e21d: Status 404 returned error can't find the container with id a9c67262d18efd93c5bae75e748e0ecaa340d831f79df1fefafbf7fd0f08e21d
Sep 15 20:48:30.128120 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:30.128079    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht" event={"ID":"3c4236ea-8bca-486c-877d-fe01d9f2d9be","Type":"ContainerStarted","Data":"a9c67262d18efd93c5bae75e748e0ecaa340d831f79df1fefafbf7fd0f08e21d"}
Sep 15 20:48:32.138296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:32.138257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht" event={"ID":"3c4236ea-8bca-486c-877d-fe01d9f2d9be","Type":"ContainerStarted","Data":"b1e2337ad2767b37abfb1681e0104e1f86f0f3fc7b796a9961391e97b4c0f741"}
Sep 15 20:48:32.138733 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:32.138344    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:32.155783 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:32.155723    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht" podStartSLOduration=1.56482242 podStartE2EDuration="3.155705472s" totalImagesPullingTime="1.590883052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:32.153997366 +0000 UTC m=+702.941121397" watchObservedRunningTime="2026-09-15 20:48:32.155705472 +0000 UTC m=+702.942829506"
Sep 15 20:48:35.107776 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:35.107746    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-d6w9v"
Sep 15 20:48:43.144551 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:43.144473    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-bc9ht"
Sep 15 20:48:45.606496 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.606459    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"]
Sep 15 20:48:45.609665 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.609647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.614582 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.614559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rp6ch\""
Sep 15 20:48:45.626854 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.626828    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"]
Sep 15 20:48:45.682001 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.681967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.682001 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.682004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw2rh\" (UniqueName: \"kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.723148 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.723105    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"]
Sep 15 20:48:45.723435 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:48:45.723410    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-sw2rh], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz" podUID="d04c6953-222b-40b1-8535-5759aa3161e8"
Sep 15 20:48:45.783249 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.783196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.783249 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.783253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw2rh\" (UniqueName: \"kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.783620 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.783600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:45.809132 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:45.809101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw2rh\" (UniqueName: \"kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:46.202619 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.202590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:46.209591 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.209570    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:46.287092 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.287057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume\") pod \"d04c6953-222b-40b1-8535-5759aa3161e8\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") "
Sep 15 20:48:46.287092 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.287101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw2rh\" (UniqueName: \"kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh\") pod \"d04c6953-222b-40b1-8535-5759aa3161e8\" (UID: \"d04c6953-222b-40b1-8535-5759aa3161e8\") "
Sep 15 20:48:46.287328 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.287303    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d04c6953-222b-40b1-8535-5759aa3161e8" (UID: "d04c6953-222b-40b1-8535-5759aa3161e8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 20:48:46.289496 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.289468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh" (OuterVolumeSpecName: "kube-api-access-sw2rh") pod "d04c6953-222b-40b1-8535-5759aa3161e8" (UID: "d04c6953-222b-40b1-8535-5759aa3161e8"). InnerVolumeSpecName "kube-api-access-sw2rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:48:46.387643 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.387601    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d04c6953-222b-40b1-8535-5759aa3161e8-extensions-socket-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:48:46.387643 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.387635    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw2rh\" (UniqueName: \"kubernetes.io/projected/d04c6953-222b-40b1-8535-5759aa3161e8-kube-api-access-sw2rh\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:48:46.689646 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.689611    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"]
Sep 15 20:48:46.696919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.696893    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"]
Sep 15 20:48:46.747576 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.747547    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:48:46.750837 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.750814    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.766486 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.766460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:48:46.790454 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.790423    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.791213 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.790653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rhv\" (UniqueName: \"kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.891110 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.891077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.891292 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.891128    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rhv\" (UniqueName: \"kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.891478 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.891460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:46.900059 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:46.900028    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rhv\" (UniqueName: \"kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv\") pod \"kuadrant-operator-controller-manager-76f8f686bf-86xth\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:47.061515 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:47.061428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:47.207075 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:47.207044    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:47.209883 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:48:47.209851    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" is forbidden: User \"system:node:ip-10-0-138-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-122.ec2.internal' and this object" podUID="d04c6953-222b-40b1-8535-5759aa3161e8" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:47.215505 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:48:47.215478    2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-jm5wz\" is forbidden: User \"system:node:ip-10-0-138-122.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-122.ec2.internal' and this object" podUID="d04c6953-222b-40b1-8535-5759aa3161e8" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-jm5wz"
Sep 15 20:48:47.340151 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:47.340123    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:48:47.342574 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:48:47.342547    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314dc432_bfe2_48ce_a5f5_f3f371644202.slice/crio-84760a53bee37013f041f9c8cbe7de34c9221b00f1dcd6f3205087809b8178c6 WatchSource:0}: Error finding container 84760a53bee37013f041f9c8cbe7de34c9221b00f1dcd6f3205087809b8178c6: Status 404 returned error can't find the container with id 84760a53bee37013f041f9c8cbe7de34c9221b00f1dcd6f3205087809b8178c6
Sep 15 20:48:47.667364 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:47.667326    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d04c6953-222b-40b1-8535-5759aa3161e8" path="/var/lib/kubelet/pods/d04c6953-222b-40b1-8535-5759aa3161e8/volumes"
Sep 15 20:48:48.212795 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:48.212765    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" event={"ID":"314dc432-bfe2-48ce-a5f5-f3f371644202","Type":"ContainerStarted","Data":"84760a53bee37013f041f9c8cbe7de34c9221b00f1dcd6f3205087809b8178c6"}
Sep 15 20:48:52.233556 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:52.233515    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" event={"ID":"314dc432-bfe2-48ce-a5f5-f3f371644202","Type":"ContainerStarted","Data":"98361d958c903ff8fbe3e92fde72061274c5b2560f5ce5d1872689d101535a36"}
Sep 15 20:48:52.234032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:52.233617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:48:52.256202 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:48:52.256155    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" podStartSLOduration=1.96739205 podStartE2EDuration="6.256139072s" totalImagesPullingTime="4.288747022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:48:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:48:52.254215571 +0000 UTC m=+723.041339603" watchObservedRunningTime="2026-09-15 20:48:52.256139072 +0000 UTC m=+723.043263101"
Sep 15 20:49:02.063038 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.062989    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:49:02.063416 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.063250    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" podUID="314dc432-bfe2-48ce-a5f5-f3f371644202" containerName="manager" containerID="cri-o://98361d958c903ff8fbe3e92fde72061274c5b2560f5ce5d1872689d101535a36" gracePeriod=10
Sep 15 20:49:02.064975 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.064954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:49:02.276871 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.276841    2579 generic.go:356] "Generic (PLEG): container finished" podID="314dc432-bfe2-48ce-a5f5-f3f371644202" containerID="98361d958c903ff8fbe3e92fde72061274c5b2560f5ce5d1872689d101535a36" exitCode=0
Sep 15 20:49:02.277060 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.276886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" event={"ID":"314dc432-bfe2-48ce-a5f5-f3f371644202","Type":"ContainerDied","Data":"98361d958c903ff8fbe3e92fde72061274c5b2560f5ce5d1872689d101535a36"}
Sep 15 20:49:02.307997 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.307974    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:49:02.423057 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.423003    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume\") pod \"314dc432-bfe2-48ce-a5f5-f3f371644202\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") "
Sep 15 20:49:02.423259 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.423073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6rhv\" (UniqueName: \"kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv\") pod \"314dc432-bfe2-48ce-a5f5-f3f371644202\" (UID: \"314dc432-bfe2-48ce-a5f5-f3f371644202\") "
Sep 15 20:49:02.423513 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.423494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "314dc432-bfe2-48ce-a5f5-f3f371644202" (UID: "314dc432-bfe2-48ce-a5f5-f3f371644202"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 20:49:02.425367 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.425349    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv" (OuterVolumeSpecName: "kube-api-access-z6rhv") pod "314dc432-bfe2-48ce-a5f5-f3f371644202" (UID: "314dc432-bfe2-48ce-a5f5-f3f371644202"). InnerVolumeSpecName "kube-api-access-z6rhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:49:02.523610 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.523572    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rhv\" (UniqueName: \"kubernetes.io/projected/314dc432-bfe2-48ce-a5f5-f3f371644202-kube-api-access-z6rhv\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:49:02.523610 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:02.523602    2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/314dc432-bfe2-48ce-a5f5-f3f371644202-extensions-socket-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:49:03.282387 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.282351    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth" event={"ID":"314dc432-bfe2-48ce-a5f5-f3f371644202","Type":"ContainerDied","Data":"84760a53bee37013f041f9c8cbe7de34c9221b00f1dcd6f3205087809b8178c6"}
Sep 15 20:49:03.282387 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.282378    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"
Sep 15 20:49:03.282873 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.282420    2579 scope.go:117] "RemoveContainer" containerID="98361d958c903ff8fbe3e92fde72061274c5b2560f5ce5d1872689d101535a36"
Sep 15 20:49:03.304242 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.304216    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:49:03.313412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.313389    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-86xth"]
Sep 15 20:49:03.667393 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:03.667361    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="314dc432-bfe2-48ce-a5f5-f3f371644202" path="/var/lib/kubelet/pods/314dc432-bfe2-48ce-a5f5-f3f371644202/volumes"
Sep 15 20:49:30.751181 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.751138    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:30.751641 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.751507    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="314dc432-bfe2-48ce-a5f5-f3f371644202" containerName="manager"
Sep 15 20:49:30.751641 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.751521    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="314dc432-bfe2-48ce-a5f5-f3f371644202" containerName="manager"
Sep 15 20:49:30.751641 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.751575    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="314dc432-bfe2-48ce-a5f5-f3f371644202" containerName="manager"
Sep 15 20:49:30.753772 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.753758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.756422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.756406    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5k97\""
Sep 15 20:49:30.756422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.756415    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 15 20:49:30.762390 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.762368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:30.853872 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.853843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:30.867704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.867665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.867838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.867729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppkdx\" (UniqueName: \"kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.969166 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.969130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.969358 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.969177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppkdx\" (UniqueName: \"kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.969787 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.969764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:30.976785 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:30.976758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppkdx\" (UniqueName: \"kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx\") pod \"limitador-limitador-8588749977-5j5c8\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") " pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:31.064204 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.064122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:31.068445 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.068423    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:49:31.070862 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.070843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.088380 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.088357    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:49:31.104581 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.104510    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:49:31.170439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.170403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4pq6\" (UniqueName: \"kubernetes.io/projected/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-kube-api-access-l4pq6\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.170439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.170439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-config-file\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.271237 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.271200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-config-file\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.271394 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.271283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4pq6\" (UniqueName: \"kubernetes.io/projected/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-kube-api-access-l4pq6\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.271996 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.271974    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-config-file\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.279766 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.279743    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4pq6\" (UniqueName: \"kubernetes.io/projected/0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e-kube-api-access-l4pq6\") pod \"limitador-limitador-84994f7ffc-klcvg\" (UID: \"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.342517 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.342459    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:31.344784 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:49:31.344755    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446643eb_64ce_406d_b0e3_2a0207d0d174.slice/crio-3edf04547df4bbb4db99138dcb79e6a5157b46d5e65ab816d7f074da3633804a WatchSource:0}: Error finding container 3edf04547df4bbb4db99138dcb79e6a5157b46d5e65ab816d7f074da3633804a: Status 404 returned error can't find the container with id 3edf04547df4bbb4db99138dcb79e6a5157b46d5e65ab816d7f074da3633804a
Sep 15 20:49:31.400717 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.400681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:31.407450 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.407423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" event={"ID":"446643eb-64ce-406d-b0e3-2a0207d0d174","Type":"ContainerStarted","Data":"3edf04547df4bbb4db99138dcb79e6a5157b46d5e65ab816d7f074da3633804a"}
Sep 15 20:49:31.671137 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:31.671106    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:49:31.674195 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:49:31.674168    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0115e4b9_ad5e_44a1_8184_3dc7dd9cbb4e.slice/crio-7e48116e1349cb587bbaca569ddf5354f8be655659171aa2fe36559b0f040566 WatchSource:0}: Error finding container 7e48116e1349cb587bbaca569ddf5354f8be655659171aa2fe36559b0f040566: Status 404 returned error can't find the container with id 7e48116e1349cb587bbaca569ddf5354f8be655659171aa2fe36559b0f040566
Sep 15 20:49:32.415057 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:32.414997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg" event={"ID":"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e","Type":"ContainerStarted","Data":"7e48116e1349cb587bbaca569ddf5354f8be655659171aa2fe36559b0f040566"}
Sep 15 20:49:33.420412 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.420376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" event={"ID":"446643eb-64ce-406d-b0e3-2a0207d0d174","Type":"ContainerStarted","Data":"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"}
Sep 15 20:49:33.420875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.420435    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:33.421843 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.421821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg" event={"ID":"0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e","Type":"ContainerStarted","Data":"01ddba74197563c8a978c764abed82787a43c2b6ef914b58d99de15f2a98cd0b"}
Sep 15 20:49:33.421968 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.421949    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:33.436230 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.436191    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" podStartSLOduration=2.077629271 podStartE2EDuration="3.4361777s" totalImagesPullingTime="1.358548429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:33.435506006 +0000 UTC m=+764.222630037" watchObservedRunningTime="2026-09-15 20:49:33.4361777 +0000 UTC m=+764.223301729"
Sep 15 20:49:33.451433 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:33.451385    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg" podStartSLOduration=1.423881621 podStartE2EDuration="2.451373075s" totalImagesPullingTime="1.027491454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:49:33.450926065 +0000 UTC m=+764.238050097" watchObservedRunningTime="2026-09-15 20:49:33.451373075 +0000 UTC m=+764.238497104"
Sep 15 20:49:44.426950 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:44.426921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-klcvg"
Sep 15 20:49:44.427473 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:44.426986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:44.485837 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:44.485809    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:44.485990 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:44.485955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" podUID="446643eb-64ce-406d-b0e3-2a0207d0d174" containerName="limitador" containerID="cri-o://1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217" gracePeriod=30
Sep 15 20:49:45.429391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.429365    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:45.474653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.474620    2579 generic.go:356] "Generic (PLEG): container finished" podID="446643eb-64ce-406d-b0e3-2a0207d0d174" containerID="1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217" exitCode=0
Sep 15 20:49:45.474798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.474658    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8"
Sep 15 20:49:45.474798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.474675    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" event={"ID":"446643eb-64ce-406d-b0e3-2a0207d0d174","Type":"ContainerDied","Data":"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"}
Sep 15 20:49:45.474798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.474725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-5j5c8" event={"ID":"446643eb-64ce-406d-b0e3-2a0207d0d174","Type":"ContainerDied","Data":"3edf04547df4bbb4db99138dcb79e6a5157b46d5e65ab816d7f074da3633804a"}
Sep 15 20:49:45.474798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.474749    2579 scope.go:117] "RemoveContainer" containerID="1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"
Sep 15 20:49:45.482777 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.482760    2579 scope.go:117] "RemoveContainer" containerID="1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"
Sep 15 20:49:45.483051 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:49:45.483029    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217\": container with ID starting with 1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217 not found: ID does not exist" containerID="1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"
Sep 15 20:49:45.483164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.483057    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217"} err="failed to get container status \"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217\": rpc error: code = NotFound desc = could not find container \"1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217\": container with ID starting with 1fd1026a12bd5195b5ddb54974135fc7f25ce8d3728d40186e49cd6189846217 not found: ID does not exist"
Sep 15 20:49:45.490419 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.490398    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppkdx\" (UniqueName: \"kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx\") pod \"446643eb-64ce-406d-b0e3-2a0207d0d174\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") "
Sep 15 20:49:45.490519 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.490461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file\") pod \"446643eb-64ce-406d-b0e3-2a0207d0d174\" (UID: \"446643eb-64ce-406d-b0e3-2a0207d0d174\") "
Sep 15 20:49:45.490753 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.490734    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file" (OuterVolumeSpecName: "config-file") pod "446643eb-64ce-406d-b0e3-2a0207d0d174" (UID: "446643eb-64ce-406d-b0e3-2a0207d0d174"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:49:45.492509 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.492485    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx" (OuterVolumeSpecName: "kube-api-access-ppkdx") pod "446643eb-64ce-406d-b0e3-2a0207d0d174" (UID: "446643eb-64ce-406d-b0e3-2a0207d0d174"). InnerVolumeSpecName "kube-api-access-ppkdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:49:45.591666 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.591591    2579 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/446643eb-64ce-406d-b0e3-2a0207d0d174-config-file\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:49:45.591666 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.591614    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppkdx\" (UniqueName: \"kubernetes.io/projected/446643eb-64ce-406d-b0e3-2a0207d0d174-kube-api-access-ppkdx\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:49:45.789777 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.789748    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:45.793532 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:45.793507    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-5j5c8"]
Sep 15 20:49:47.668324 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:49:47.668292    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="446643eb-64ce-406d-b0e3-2a0207d0d174" path="/var/lib/kubelet/pods/446643eb-64ce-406d-b0e3-2a0207d0d174/volumes"
Sep 15 20:50:07.768944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.768898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:07.769422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.769403    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="446643eb-64ce-406d-b0e3-2a0207d0d174" containerName="limitador"
Sep 15 20:50:07.769464 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.769428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="446643eb-64ce-406d-b0e3-2a0207d0d174" containerName="limitador"
Sep 15 20:50:07.769524 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.769515    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="446643eb-64ce-406d-b0e3-2a0207d0d174" containerName="limitador"
Sep 15 20:50:07.773098 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.773079    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.775470 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.775452    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 20:50:07.775925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.775907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-shjx4\""
Sep 15 20:50:07.776037 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.775907    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 20:50:07.776037 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.775968    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 20:50:07.781687 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.781186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:07.871794 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.871767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.871944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.871824    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.871944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.871849    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pbh\" (UniqueName: \"kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.950740 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.950708    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 20:50:07.953050 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.953035    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:07.964385 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.964366    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 20:50:07.972446 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.972423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.972578 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.972477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.972643 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:50:07.972627    2579 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 15 20:50:07.972700 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.972662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pbh\" (UniqueName: \"kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.972700 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:50:07.972677    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs podName:3c1215ee-fb3b-488d-a2b3-013e387f24a5 nodeName:}" failed. No retries permitted until 2026-09-15 20:50:08.472659577 +0000 UTC m=+799.259783597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs") pod "maas-controller-85cbd9b-tnn9s" (UID: "3c1215ee-fb3b-488d-a2b3-013e387f24a5") : secret "maas-controller-metrics-tls" not found
Sep 15 20:50:07.975693 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.975662    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:07.984860 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:07.984837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pbh\" (UniqueName: \"kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:08.073478 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.073382    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.073478 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.073444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.073669 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.073503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwbqr\" (UniqueName: \"kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.174434 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.174397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.174434 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.174443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.174700 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.174480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwbqr\" (UniqueName: \"kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.177198 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.177168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.177355 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.177168    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.183401 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.183381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwbqr\" (UniqueName: \"kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr\") pod \"maas-controller-68d9b8fcfd-smxg4\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") " pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.262982 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.262945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:08.476890 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.476839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:08.479323 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.479295    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") pod \"maas-controller-85cbd9b-tnn9s\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") " pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:08.534481 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.534460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 20:50:08.536579 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:50:08.536554    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed9c28d_ab12_4def_86c4_0e29b531acc1.slice/crio-0f484d4855b1a7e28a5db6ac696f97ff32facbe39459bbdc48a10b0ca6d25ec6 WatchSource:0}: Error finding container 0f484d4855b1a7e28a5db6ac696f97ff32facbe39459bbdc48a10b0ca6d25ec6: Status 404 returned error can't find the container with id 0f484d4855b1a7e28a5db6ac696f97ff32facbe39459bbdc48a10b0ca6d25ec6
Sep 15 20:50:08.581084 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.581057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" event={"ID":"eed9c28d-ab12-4def-86c4-0e29b531acc1","Type":"ContainerStarted","Data":"0f484d4855b1a7e28a5db6ac696f97ff32facbe39459bbdc48a10b0ca6d25ec6"}
Sep 15 20:50:08.686603 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.686568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:08.955819 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:08.955796    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:08.958301 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:50:08.958279    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c1215ee_fb3b_488d_a2b3_013e387f24a5.slice/crio-76b11284af19d662e8bd4327bd4b944bc5c1a4b055bd900498d654acbbb580a3 WatchSource:0}: Error finding container 76b11284af19d662e8bd4327bd4b944bc5c1a4b055bd900498d654acbbb580a3: Status 404 returned error can't find the container with id 76b11284af19d662e8bd4327bd4b944bc5c1a4b055bd900498d654acbbb580a3
Sep 15 20:50:09.592288 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:09.592246    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cbd9b-tnn9s" event={"ID":"3c1215ee-fb3b-488d-a2b3-013e387f24a5","Type":"ContainerStarted","Data":"76b11284af19d662e8bd4327bd4b944bc5c1a4b055bd900498d654acbbb580a3"}
Sep 15 20:50:11.605575 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.605472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" event={"ID":"eed9c28d-ab12-4def-86c4-0e29b531acc1","Type":"ContainerStarted","Data":"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"}
Sep 15 20:50:11.605575 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.605537    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:11.607377 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.607336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cbd9b-tnn9s" event={"ID":"3c1215ee-fb3b-488d-a2b3-013e387f24a5","Type":"ContainerStarted","Data":"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"}
Sep 15 20:50:11.607486 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.607413    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:11.622256 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.622213    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" podStartSLOduration=1.997920717 podStartE2EDuration="4.622200789s" totalImagesPullingTime="2.624280072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:11.621140437 +0000 UTC m=+802.408264476" watchObservedRunningTime="2026-09-15 20:50:11.622200789 +0000 UTC m=+802.409324816"
Sep 15 20:50:11.640032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:11.639981    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-85cbd9b-tnn9s" podStartSLOduration=2.428282923 podStartE2EDuration="4.639966628s" totalImagesPullingTime="2.211683705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:11.638237899 +0000 UTC m=+802.425361928" watchObservedRunningTime="2026-09-15 20:50:11.639966628 +0000 UTC m=+802.427090658"
Sep 15 20:50:22.614530 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:22.614498    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 20:50:22.614996 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:22.614569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:22.672085 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:22.671887    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:22.672422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:22.672391    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-85cbd9b-tnn9s" podUID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" containerName="manager" containerID="cri-o://f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7" gracePeriod=30
Sep 15 20:50:22.919432 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:22.919409    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:23.000495 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.000461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert\") pod \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") "
Sep 15 20:50:23.000657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.000547    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") pod \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") "
Sep 15 20:50:23.000657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.000593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9pbh\" (UniqueName: \"kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh\") pod \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\" (UID: \"3c1215ee-fb3b-488d-a2b3-013e387f24a5\") "
Sep 15 20:50:23.002776 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.002741    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh" (OuterVolumeSpecName: "kube-api-access-p9pbh") pod "3c1215ee-fb3b-488d-a2b3-013e387f24a5" (UID: "3c1215ee-fb3b-488d-a2b3-013e387f24a5"). InnerVolumeSpecName "kube-api-access-p9pbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:50:23.002776 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.002762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c1215ee-fb3b-488d-a2b3-013e387f24a5" (UID: "3c1215ee-fb3b-488d-a2b3-013e387f24a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:50:23.002919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.002783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3c1215ee-fb3b-488d-a2b3-013e387f24a5" (UID: "3c1215ee-fb3b-488d-a2b3-013e387f24a5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 20:50:23.101695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.101662    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-webhook-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:50:23.101695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.101688    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c1215ee-fb3b-488d-a2b3-013e387f24a5-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:50:23.101695 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.101700    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pbh\" (UniqueName: \"kubernetes.io/projected/3c1215ee-fb3b-488d-a2b3-013e387f24a5-kube-api-access-p9pbh\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:50:23.661638 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.661608    2579 generic.go:356] "Generic (PLEG): container finished" podID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" containerID="f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7" exitCode=0
Sep 15 20:50:23.662147 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.661661    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-85cbd9b-tnn9s"
Sep 15 20:50:23.662147 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.661698    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cbd9b-tnn9s" event={"ID":"3c1215ee-fb3b-488d-a2b3-013e387f24a5","Type":"ContainerDied","Data":"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"}
Sep 15 20:50:23.662147 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.661821    2579 scope.go:117] "RemoveContainer" containerID="f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"
Sep 15 20:50:23.673167 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.670968    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-85cbd9b-tnn9s" event={"ID":"3c1215ee-fb3b-488d-a2b3-013e387f24a5","Type":"ContainerDied","Data":"76b11284af19d662e8bd4327bd4b944bc5c1a4b055bd900498d654acbbb580a3"}
Sep 15 20:50:23.673631 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.673223    2579 scope.go:117] "RemoveContainer" containerID="f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"
Sep 15 20:50:23.673684 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:50:23.673611    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7\": container with ID starting with f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7 not found: ID does not exist" containerID="f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"
Sep 15 20:50:23.673684 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.673660    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7"} err="failed to get container status \"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7\": rpc error: code = NotFound desc = could not find container \"f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7\": container with ID starting with f2ac380a017ec4491d12b0d32fc6ca7e39e5a92926ee9d8122bb20725a6458d7 not found: ID does not exist"
Sep 15 20:50:23.689399 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.689368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:23.692214 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:23.692194    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-85cbd9b-tnn9s"]
Sep 15 20:50:25.668047 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:25.667995    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" path="/var/lib/kubelet/pods/3c1215ee-fb3b-488d-a2b3-013e387f24a5/volumes"
Sep 15 20:50:44.427587 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.427551    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 20:50:44.428082 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.428066    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" containerName="manager"
Sep 15 20:50:44.428132 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.428085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" containerName="manager"
Sep 15 20:50:44.428170 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.428164    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1215ee-fb3b-488d-a2b3-013e387f24a5" containerName="manager"
Sep 15 20:50:44.434447 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.434426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.437197 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.437175    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 15 20:50:44.437309 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.437180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-crdcx\""
Sep 15 20:50:44.437505 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.437491    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 15 20:50:44.438417 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.438405    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 15 20:50:44.438491 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.438463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 15 20:50:44.441815 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.441793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 20:50:44.466375 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.466337    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.466526 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.466383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.466526 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.466455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsrg8\" (UniqueName: \"kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.466614 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.466523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.567019 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.566974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.567212 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.567070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.567212 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.567100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.567212 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.567146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsrg8\" (UniqueName: \"kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.567710 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.567679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.569703 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.569683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.569928 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.569908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.574857 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.574832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsrg8\" (UniqueName: \"kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8\") pod \"maas-api-746c646f77-884fh\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") " pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.747432 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.747357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:44.974303 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.974269    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"]
Sep 15 20:50:44.976623 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.976609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:44.979891 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.979876    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l5l9x\""
Sep 15 20:50:44.980206 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:44.980194    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 15 20:50:45.009046 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.008929    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"]
Sep 15 20:50:45.068305 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.068277    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 20:50:45.070388 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:50:45.070359    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa90bf3_e629_4b15_90a3_5799acb35a6e.slice/crio-7a93eec1a2cfb7cf948551ab839d8005bffaf4a1d53c60cb6380ac0427571a66 WatchSource:0}: Error finding container 7a93eec1a2cfb7cf948551ab839d8005bffaf4a1d53c60cb6380ac0427571a66: Status 404 returned error can't find the container with id 7a93eec1a2cfb7cf948551ab839d8005bffaf4a1d53c60cb6380ac0427571a66
Sep 15 20:50:45.070740 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.070718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttm6d\" (UniqueName: \"kubernetes.io/projected/80b73a39-dc35-4151-b2f5-23bfbf56cadb-kube-api-access-ttm6d\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.070784 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.070770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b73a39-dc35-4151-b2f5-23bfbf56cadb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.171378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.171337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttm6d\" (UniqueName: \"kubernetes.io/projected/80b73a39-dc35-4151-b2f5-23bfbf56cadb-kube-api-access-ttm6d\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.171378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.171387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b73a39-dc35-4151-b2f5-23bfbf56cadb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.172032 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.171997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b73a39-dc35-4151-b2f5-23bfbf56cadb-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.179307 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.179279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttm6d\" (UniqueName: \"kubernetes.io/projected/80b73a39-dc35-4151-b2f5-23bfbf56cadb-kube-api-access-ttm6d\") pod \"payload-processing-5b54b7fbb7-bn6qq\" (UID: \"80b73a39-dc35-4151-b2f5-23bfbf56cadb\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.287290 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.287213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:45.555539 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.555334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"]
Sep 15 20:50:45.558142 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:50:45.558116    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b73a39_dc35_4151_b2f5_23bfbf56cadb.slice/crio-c8e3722a985901463cd4ab8cf0d2c70f3a281b7d3e8a985420f75d9e0c516f8d WatchSource:0}: Error finding container c8e3722a985901463cd4ab8cf0d2c70f3a281b7d3e8a985420f75d9e0c516f8d: Status 404 returned error can't find the container with id c8e3722a985901463cd4ab8cf0d2c70f3a281b7d3e8a985420f75d9e0c516f8d
Sep 15 20:50:45.714956 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.714925    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:50:45.759165 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.759117    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" event={"ID":"5fa90bf3-e629-4b15-90a3-5799acb35a6e","Type":"ContainerStarted","Data":"7a93eec1a2cfb7cf948551ab839d8005bffaf4a1d53c60cb6380ac0427571a66"}
Sep 15 20:50:45.760644 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:45.760619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq" event={"ID":"80b73a39-dc35-4151-b2f5-23bfbf56cadb","Type":"ContainerStarted","Data":"c8e3722a985901463cd4ab8cf0d2c70f3a281b7d3e8a985420f75d9e0c516f8d"}
Sep 15 20:50:47.770777 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.770743    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" event={"ID":"5fa90bf3-e629-4b15-90a3-5799acb35a6e","Type":"ContainerStarted","Data":"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"}
Sep 15 20:50:47.771250 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.770831    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:47.772375 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.772354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq" event={"ID":"80b73a39-dc35-4151-b2f5-23bfbf56cadb","Type":"ContainerStarted","Data":"504ffe4164ee1a34e679dfb03af717d84b782d8b2d9e9507e1e84fd39ac28367"}
Sep 15 20:50:47.772539 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.772480    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:50:47.789034 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.787454    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" podStartSLOduration=1.87752564 podStartE2EDuration="3.787437591s" totalImagesPullingTime="1.909911951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:47.786479 +0000 UTC m=+838.573603044" watchObservedRunningTime="2026-09-15 20:50:47.787437591 +0000 UTC m=+838.574561620"
Sep 15 20:50:47.805302 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:47.805257    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq" podStartSLOduration=2.383068033 podStartE2EDuration="3.80524471s" totalImagesPullingTime="1.422176677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:50:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:50:47.804140567 +0000 UTC m=+838.591264596" watchObservedRunningTime="2026-09-15 20:50:47.80524471 +0000 UTC m=+838.592368738"
Sep 15 20:50:53.782829 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:53.782794    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 20:50:58.779389 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:50:58.779363    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-bn6qq"
Sep 15 20:51:01.009718 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:01.009681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:13.458295 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.458261    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"]
Sep 15 20:51:13.462940 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.462911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.466698 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.466676    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 15 20:51:13.466813 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.466676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 15 20:51:13.466813 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.466676    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bwfmg\""
Sep 15 20:51:13.466921 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.466763    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 15 20:51:13.471853 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.471833    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"]
Sep 15 20:51:13.611475 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.611475 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.611657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.611657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4vzk\" (UniqueName: \"kubernetes.io/projected/f20fac15-e5b9-4988-b162-2328087e0946-kube-api-access-x4vzk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.611657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fac15-e5b9-4988-b162-2328087e0946-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.611657 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.611651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712205 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712205 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4vzk\" (UniqueName: \"kubernetes.io/projected/f20fac15-e5b9-4988-b162-2328087e0946-kube-api-access-x4vzk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fac15-e5b9-4988-b162-2328087e0946-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712423 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712291    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712647 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712710 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712652    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.712710 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.712670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.714441 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.714425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f20fac15-e5b9-4988-b162-2328087e0946-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.714710 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.714693    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f20fac15-e5b9-4988-b162-2328087e0946-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.722779 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.722758    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4vzk\" (UniqueName: \"kubernetes.io/projected/f20fac15-e5b9-4988-b162-2328087e0946-kube-api-access-x4vzk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4868\" (UID: \"f20fac15-e5b9-4988-b162-2328087e0946\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:13.773692 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:13.773659    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:14.045220 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:14.045196    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"]
Sep 15 20:51:14.047436 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:51:14.047407    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20fac15_e5b9_4988_b162_2328087e0946.slice/crio-21a6811437ae8623f624249fca8fd053299148a6a276ead4128543afca530ebc WatchSource:0}: Error finding container 21a6811437ae8623f624249fca8fd053299148a6a276ead4128543afca530ebc: Status 404 returned error can't find the container with id 21a6811437ae8623f624249fca8fd053299148a6a276ead4128543afca530ebc
Sep 15 20:51:14.470659 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:14.470619    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:14.892151 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:14.892109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868" event={"ID":"f20fac15-e5b9-4988-b162-2328087e0946","Type":"ContainerStarted","Data":"21a6811437ae8623f624249fca8fd053299148a6a276ead4128543afca530ebc"}
Sep 15 20:51:15.897531 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:15.897497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868" event={"ID":"f20fac15-e5b9-4988-b162-2328087e0946","Type":"ContainerStarted","Data":"a004d0dff6c6e5d40b8b4e77f9c9222a8c27b7a3592d739e5e5ab08c2c710bf9"}
Sep 15 20:51:15.914068 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:15.914025    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868" podStartSLOduration=1.927847225 podStartE2EDuration="2.913991888s" totalImagesPullingTime="986.144663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:15.912373086 +0000 UTC m=+866.699497117" watchObservedRunningTime="2026-09-15 20:51:15.913991888 +0000 UTC m=+866.701115916"
Sep 15 20:51:23.774296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:23.774246    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:23.774780 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:23.774335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:23.787240 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:23.787200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:23.942953 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:23.942928    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4868"
Sep 15 20:51:24.265944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:24.265910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:39.670150 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.670121    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"]
Sep 15 20:51:39.675725 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.675706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.679176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.679157    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 15 20:51:39.682784 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.682764    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"]
Sep 15 20:51:39.742040 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.741989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de42155-f36a-40ef-92f8-6849d66fc638-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.742221 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.742085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.742221 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.742165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.742221 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.742193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.742394 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.742259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.742394 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.742289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtwv6\" (UniqueName: \"kubernetes.io/projected/7de42155-f36a-40ef-92f8-6849d66fc638-kube-api-access-rtwv6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843083 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843288 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843288 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843146    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843419 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843297    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843419 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843334    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtwv6\" (UniqueName: \"kubernetes.io/projected/7de42155-f36a-40ef-92f8-6849d66fc638-kube-api-access-rtwv6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843419 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843370    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de42155-f36a-40ef-92f8-6849d66fc638-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843576 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843435    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843576 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.843654 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.843618    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.845498 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.845471    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7de42155-f36a-40ef-92f8-6849d66fc638-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.846037 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.845997    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de42155-f36a-40ef-92f8-6849d66fc638-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:39.851572 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:39.851547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtwv6\" (UniqueName: \"kubernetes.io/projected/7de42155-f36a-40ef-92f8-6849d66fc638-kube-api-access-rtwv6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k4969\" (UID: \"7de42155-f36a-40ef-92f8-6849d66fc638\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:40.011570 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:40.011476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:40.290751 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:40.290651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"]
Sep 15 20:51:40.292981 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:51:40.292956    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de42155_f36a_40ef_92f8_6849d66fc638.slice/crio-b50e1b52a380b8b6584d0f65813bd39628b845bfc3ca4c9904afe41cbd46ea7e WatchSource:0}: Error finding container b50e1b52a380b8b6584d0f65813bd39628b845bfc3ca4c9904afe41cbd46ea7e: Status 404 returned error can't find the container with id b50e1b52a380b8b6584d0f65813bd39628b845bfc3ca4c9904afe41cbd46ea7e
Sep 15 20:51:41.009521 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:41.009442    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969" event={"ID":"7de42155-f36a-40ef-92f8-6849d66fc638","Type":"ContainerStarted","Data":"da5f8f81db2f1efa0741a8b318dec8683c59b6b47d3f3f43bbbb9be72cfbbbec"}
Sep 15 20:51:41.009521 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:41.009494    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969" event={"ID":"7de42155-f36a-40ef-92f8-6849d66fc638","Type":"ContainerStarted","Data":"b50e1b52a380b8b6584d0f65813bd39628b845bfc3ca4c9904afe41cbd46ea7e"}
Sep 15 20:51:41.027674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:41.027627    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969" podStartSLOduration=1.838394493 podStartE2EDuration="2.02761199s" totalImagesPullingTime="189.217497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:41.026966175 +0000 UTC m=+891.814090205" watchObservedRunningTime="2026-09-15 20:51:41.02761199 +0000 UTC m=+891.814736067"
Sep 15 20:51:41.072989 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:41.072957    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:46.063759 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.063724    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"]
Sep 15 20:51:46.067613 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.067590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.070439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.070417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 15 20:51:46.076589 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.076565    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"]
Sep 15 20:51:46.097811 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.097788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.097925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.097816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.097925 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.097832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8626f72c-a7d6-4861-850e-ff02ad41aa51-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.097993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.097927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.097993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.097973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.098104 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.098002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9r9f\" (UniqueName: \"kubernetes.io/projected/8626f72c-a7d6-4861-850e-ff02ad41aa51-kube-api-access-v9r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.198970 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.198937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.198984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9r9f\" (UniqueName: \"kubernetes.io/projected/8626f72c-a7d6-4861-850e-ff02ad41aa51-kube-api-access-v9r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199088    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199164 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199134    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8626f72c-a7d6-4861-850e-ff02ad41aa51-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199526 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199473    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.199566 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.199523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.201506 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.201488    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8626f72c-a7d6-4861-850e-ff02ad41aa51-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.201746 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.201730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8626f72c-a7d6-4861-850e-ff02ad41aa51-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.208555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.208533    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9r9f\" (UniqueName: \"kubernetes.io/projected/8626f72c-a7d6-4861-850e-ff02ad41aa51-kube-api-access-v9r9f\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t\" (UID: \"8626f72c-a7d6-4861-850e-ff02ad41aa51\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.384655 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.384468    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:46.655950 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:46.655921    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"]
Sep 15 20:51:46.657837 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:51:46.657811    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8626f72c_a7d6_4861_850e_ff02ad41aa51.slice/crio-a7364cf2c4791a381fe0d104bb346a13b1ba86919f0f3e2071a272c436c5eef8 WatchSource:0}: Error finding container a7364cf2c4791a381fe0d104bb346a13b1ba86919f0f3e2071a272c436c5eef8: Status 404 returned error can't find the container with id a7364cf2c4791a381fe0d104bb346a13b1ba86919f0f3e2071a272c436c5eef8
Sep 15 20:51:47.037794 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:47.037762    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t" event={"ID":"8626f72c-a7d6-4861-850e-ff02ad41aa51","Type":"ContainerStarted","Data":"55139b846b1594a165b772d681e8f9f87db0d055bb6da5c375cb232bdc62a925"}
Sep 15 20:51:47.037965 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:47.037821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t" event={"ID":"8626f72c-a7d6-4861-850e-ff02ad41aa51","Type":"ContainerStarted","Data":"a7364cf2c4791a381fe0d104bb346a13b1ba86919f0f3e2071a272c436c5eef8"}
Sep 15 20:51:47.055508 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:47.055462    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t" podStartSLOduration=0.857382129 podStartE2EDuration="1.055446998s" totalImagesPullingTime="198.064869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:51:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:51:47.053744176 +0000 UTC m=+897.840868206" watchObservedRunningTime="2026-09-15 20:51:47.055446998 +0000 UTC m=+897.842571026"
Sep 15 20:51:47.564171 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:47.564138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:50.012504 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:50.012447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:50.013053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:50.012531    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:50.025445 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:50.025420    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:50.063747 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:50.063714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k4969"
Sep 15 20:51:55.268520 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:55.268483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:51:56.384941 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:56.384901    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:56.385392 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:56.384975    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:56.397339 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:56.397313    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:51:57.095965 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:51:57.095929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t"
Sep 15 20:52:06.176554 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:52:06.176520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:54:13.188745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.188711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:54:13.427256 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.427223    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 20:54:13.430651 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.430629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.433184 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.433161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-d5b5b9-serving-cert\""
Sep 15 20:54:13.441338 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.441282    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 20:54:13.563436 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.563393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.563436 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.563431    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.563688 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.563451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.563688 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.563519    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.664820 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.664782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.665039 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.664868    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.665039 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.664895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.665039 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.664934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.665631 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.665605    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.667601 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.667577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.667772 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.667753    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.673203 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.673183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh\") pod \"maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:13.744165 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:13.744082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:14.023955 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.023934    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 20:54:14.026526 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:14.026497    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0be28f_5dd6_4813_949d_ca07fa5680a8.slice/crio-ac0a4b0105fb4c96cd1a5aef85df2bcf66f4006ace3f54878ea67ecbe5f65069 WatchSource:0}: Error finding container ac0a4b0105fb4c96cd1a5aef85df2bcf66f4006ace3f54878ea67ecbe5f65069: Status 404 returned error can't find the container with id ac0a4b0105fb4c96cd1a5aef85df2bcf66f4006ace3f54878ea67ecbe5f65069
Sep 15 20:54:14.027944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.027925    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 20:54:14.123645 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.123604    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 20:54:14.127978 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.127963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.130577 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.130556    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d5b5b9-dockercfg-8vc62\""
Sep 15 20:54:14.130685 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.130577    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d5b5b9\""
Sep 15 20:54:14.136510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.136489    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 20:54:14.270918 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.270864    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.270918 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.270907    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwp6h\" (UniqueName: \"kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.371738 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.371703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.371917 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.371745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwp6h\" (UniqueName: \"kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.372446 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.372427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.384259 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.384229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwp6h\" (UniqueName: \"kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h\") pod \"payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.439644 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.439607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:14.709673 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.709639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 20:54:14.712303 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:14.712278    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bd40cc_61f9_4001_acea_1b0cd11dc96d.slice/crio-f15f9d8ada652e9de0299192967b441a5d8138f5a99c9e72671b30961b09dbb7 WatchSource:0}: Error finding container f15f9d8ada652e9de0299192967b441a5d8138f5a99c9e72671b30961b09dbb7: Status 404 returned error can't find the container with id f15f9d8ada652e9de0299192967b441a5d8138f5a99c9e72671b30961b09dbb7
Sep 15 20:54:14.722707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.722671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" event={"ID":"1a0be28f-5dd6-4813-949d-ca07fa5680a8","Type":"ContainerStarted","Data":"a468c3c9257374c875337462c5cdeb34ee9116557f4cd2943df79d099a088244"}
Sep 15 20:54:14.722813 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.722733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" event={"ID":"1a0be28f-5dd6-4813-949d-ca07fa5680a8","Type":"ContainerStarted","Data":"ac0a4b0105fb4c96cd1a5aef85df2bcf66f4006ace3f54878ea67ecbe5f65069"}
Sep 15 20:54:14.722813 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.722760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:14.723878 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.723855    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" event={"ID":"55bd40cc-61f9-4001-acea-1b0cd11dc96d","Type":"ContainerStarted","Data":"f15f9d8ada652e9de0299192967b441a5d8138f5a99c9e72671b30961b09dbb7"}
Sep 15 20:54:14.738575 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:14.738537    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" podStartSLOduration=1.159692459 podStartE2EDuration="1.738522538s" totalImagesPullingTime="578.830079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:14.737314928 +0000 UTC m=+1045.524438958" watchObservedRunningTime="2026-09-15 20:54:14.738522538 +0000 UTC m=+1045.525646566"
Sep 15 20:54:15.186461 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.186424    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 20:54:15.189933 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.189908    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.192560 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.192534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-16bde4-serving-cert\""
Sep 15 20:54:15.202859 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.202769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 20:54:15.280966 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.280928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.281457 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.281044    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.281457 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.281063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.281457 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.281100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxmwb\" (UniqueName: \"kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.382582 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.382546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.382767 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.382628    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.382767 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.382654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.382767 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.382698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxmwb\" (UniqueName: \"kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.383468 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.383401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.385232 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.385203    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.385430 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.385411    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.390413 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.390396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxmwb\" (UniqueName: \"kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb\") pod \"maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.505801 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.505772    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:15.731133 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.730978    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" event={"ID":"55bd40cc-61f9-4001-acea-1b0cd11dc96d","Type":"ContainerStarted","Data":"31d2e76331b195fc053a98dfb0961065d8de610eefa36876e119e4d6ce8908fa"}
Sep 15 20:54:15.731308 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.731248    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:15.748886 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.748836    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" podStartSLOduration=1.7488219200000001 podStartE2EDuration="1.74882192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:15.745829953 +0000 UTC m=+1046.532953982" watchObservedRunningTime="2026-09-15 20:54:15.74882192 +0000 UTC m=+1046.535945949"
Sep 15 20:54:15.778444 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.778409    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 20:54:15.782778 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.782755    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.786241 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.786204    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-16bde4-dockercfg-pfnq8\""
Sep 15 20:54:15.786422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.786399    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-16bde4\""
Sep 15 20:54:15.787296 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.787184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 20:54:15.791855 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.791814    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 20:54:15.887760 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.887600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgp6\" (UniqueName: \"kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.887760 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.887657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.988916 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.988827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgp6\" (UniqueName: \"kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.988916 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.988879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.989617 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.989587    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:15.998902 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:15.998876    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgp6\" (UniqueName: \"kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6\") pod \"payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:16.105381 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.105350    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:16.383440 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.383416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 20:54:16.386293 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:16.386264    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74366d23_b564_4e72_b80e_694bd75ab64b.slice/crio-b5c828d1cc54cd8c3b82919b34368564923a9bc244ae095eede5d67641121538 WatchSource:0}: Error finding container b5c828d1cc54cd8c3b82919b34368564923a9bc244ae095eede5d67641121538: Status 404 returned error can't find the container with id b5c828d1cc54cd8c3b82919b34368564923a9bc244ae095eede5d67641121538
Sep 15 20:54:16.671321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.671245    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 20:54:16.675239 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.675215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.677649 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.677623    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-3f1c93\""
Sep 15 20:54:16.677849 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.677833    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-3f1c93-dockercfg-p2rht\""
Sep 15 20:54:16.684842 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.684817    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 20:54:16.739372 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.739337    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" event={"ID":"74366d23-b564-4e72-b80e-694bd75ab64b","Type":"ContainerStarted","Data":"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"}
Sep 15 20:54:16.739544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.739394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" event={"ID":"74366d23-b564-4e72-b80e-694bd75ab64b","Type":"ContainerStarted","Data":"b5c828d1cc54cd8c3b82919b34368564923a9bc244ae095eede5d67641121538"}
Sep 15 20:54:16.739544 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.739508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:16.741066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.741038    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" event={"ID":"46a81999-7bf8-43ff-b62c-8d43046805e2","Type":"ContainerStarted","Data":"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"}
Sep 15 20:54:16.741201 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.741088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" event={"ID":"46a81999-7bf8-43ff-b62c-8d43046805e2","Type":"ContainerStarted","Data":"d16658a575cda8ce39edd369736a4af76c049c11fa83686b26a3bc06535f6d4d"}
Sep 15 20:54:16.758061 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.757974    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" podStartSLOduration=1.757953891 podStartE2EDuration="1.757953891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:16.75679508 +0000 UTC m=+1047.543919110" watchObservedRunningTime="2026-09-15 20:54:16.757953891 +0000 UTC m=+1047.545077922"
Sep 15 20:54:16.781730 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.781671    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" podStartSLOduration=1.129844587 podStartE2EDuration="1.781646467s" totalImagesPullingTime="651.80188ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:16.77960838 +0000 UTC m=+1047.566732421" watchObservedRunningTime="2026-09-15 20:54:16.781646467 +0000 UTC m=+1047.568770497"
Sep 15 20:54:16.796388 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.796357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97rrm\" (UniqueName: \"kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.796554 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.796416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.897162 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.897126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97rrm\" (UniqueName: \"kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.897329 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.897201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.898028 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.897977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.906134 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.906113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97rrm\" (UniqueName: \"kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm\") pod \"payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:16.991847 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:16.991763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:17.269421 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.269395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 20:54:17.270293 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:17.270263    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16c874b_803e_4e91_97dc_de9dc10a7d98.slice/crio-31ba2f7afc0910eafe000d4d6c139d5856f5c0de183dbf9ef1bbf031d22e2e7f WatchSource:0}: Error finding container 31ba2f7afc0910eafe000d4d6c139d5856f5c0de183dbf9ef1bbf031d22e2e7f: Status 404 returned error can't find the container with id 31ba2f7afc0910eafe000d4d6c139d5856f5c0de183dbf9ef1bbf031d22e2e7f
Sep 15 20:54:17.750742 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.750701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" event={"ID":"d16c874b-803e-4e91-97dc-de9dc10a7d98","Type":"ContainerStarted","Data":"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"}
Sep 15 20:54:17.751211 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.750792    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" event={"ID":"d16c874b-803e-4e91-97dc-de9dc10a7d98","Type":"ContainerStarted","Data":"31ba2f7afc0910eafe000d4d6c139d5856f5c0de183dbf9ef1bbf031d22e2e7f"}
Sep 15 20:54:17.751211 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.750822    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:17.751347 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.751324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:17.768335 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:17.768283    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" podStartSLOduration=1.768263181 podStartE2EDuration="1.768263181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:17.766126185 +0000 UTC m=+1048.553250215" watchObservedRunningTime="2026-09-15 20:54:17.768263181 +0000 UTC m=+1048.555387213"
Sep 15 20:54:20.736526 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:20.736495    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 20:54:23.759928 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:23.759896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 20:54:24.031886 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.031809    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:54:24.035562 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.035542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.038271 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.038249    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b9ff68\""
Sep 15 20:54:24.038369 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.038270    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b9ff68-dockercfg-bbc28\""
Sep 15 20:54:24.045450 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.045426    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:54:24.069157 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.069129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.069278 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.069225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8n5z\" (UniqueName: \"kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.170440 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.170401    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8n5z\" (UniqueName: \"kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.170626 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.170504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.171154 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.171136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.178126 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.178106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8n5z\" (UniqueName: \"kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z\") pod \"payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") " pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.349257 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.349165    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.621669 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.621636    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:54:24.624946 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:24.624922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29d509d_c73a_42ae_8947_8696b9ac7d85.slice/crio-142ae168489cd54909ab773b9ce516068cf41c0ffd284ed608910ada218ba1f3 WatchSource:0}: Error finding container 142ae168489cd54909ab773b9ce516068cf41c0ffd284ed608910ada218ba1f3: Status 404 returned error can't find the container with id 142ae168489cd54909ab773b9ce516068cf41c0ffd284ed608910ada218ba1f3
Sep 15 20:54:24.790451 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.790415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" event={"ID":"e29d509d-c73a-42ae-8947-8696b9ac7d85","Type":"ContainerStarted","Data":"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"}
Sep 15 20:54:24.790865 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.790473    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" event={"ID":"e29d509d-c73a-42ae-8947-8696b9ac7d85","Type":"ContainerStarted","Data":"142ae168489cd54909ab773b9ce516068cf41c0ffd284ed608910ada218ba1f3"}
Sep 15 20:54:24.790865 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.790549    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:24.806633 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:24.806571    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" podStartSLOduration=0.806554206 podStartE2EDuration="806.554206ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:24.804858822 +0000 UTC m=+1055.591982864" watchObservedRunningTime="2026-09-15 20:54:24.806554206 +0000 UTC m=+1055.593678234"
Sep 15 20:54:25.722454 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.722418    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:25.725959 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.725944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.728556 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.728534    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ded67cd1-dockercfg-bjs24\""
Sep 15 20:54:25.728655 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.728544    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ded67cd1\""
Sep 15 20:54:25.736445 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.736318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:25.780537 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.780497    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l77k\" (UniqueName: \"kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.780537 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.780536    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.883270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.881648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l77k\" (UniqueName: \"kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.883270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.881707    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.883270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.883222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:25.891462 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:25.891431    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l77k\" (UniqueName: \"kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k\") pod \"payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:26.038426 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.038342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:26.315921 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.315849    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:26.319209 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:26.319185    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a2a161_b0d8_4555_81bf_8121cea63849.slice/crio-51ce60a640a9d2d1826c952275d33a2cc60cd1c84d95535b620341035e6c6b11 WatchSource:0}: Error finding container 51ce60a640a9d2d1826c952275d33a2cc60cd1c84d95535b620341035e6c6b11: Status 404 returned error can't find the container with id 51ce60a640a9d2d1826c952275d33a2cc60cd1c84d95535b620341035e6c6b11
Sep 15 20:54:26.743482 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.743451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 20:54:26.803862 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.803827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" event={"ID":"86a2a161-b0d8-4555-81bf-8121cea63849","Type":"ContainerStarted","Data":"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"}
Sep 15 20:54:26.804066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.803885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" event={"ID":"86a2a161-b0d8-4555-81bf-8121cea63849","Type":"ContainerStarted","Data":"51ce60a640a9d2d1826c952275d33a2cc60cd1c84d95535b620341035e6c6b11"}
Sep 15 20:54:26.804066 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.803934    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:26.820052 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:26.819966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" podStartSLOduration=1.819946055 podStartE2EDuration="1.819946055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:26.818413207 +0000 UTC m=+1057.605537237" watchObservedRunningTime="2026-09-15 20:54:26.819946055 +0000 UTC m=+1057.607070088"
Sep 15 20:54:27.753630 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:27.753602    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 20:54:28.757522 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:28.757496    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 20:54:31.024770 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.024737    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 20:54:31.028460 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.028444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.031330 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.031304    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-988a09\""
Sep 15 20:54:31.031463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.031321    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-988a09-dockercfg-2vccj\""
Sep 15 20:54:31.037299 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.037270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 20:54:31.132291 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.132246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct6zp\" (UniqueName: \"kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.132460 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.132370    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.233254 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.233222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.233436 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.233288    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct6zp\" (UniqueName: \"kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.233900 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.233880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.240945 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.240921    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct6zp\" (UniqueName: \"kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp\") pod \"payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.340507 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.340411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.648235 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.648211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 20:54:31.649753 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:54:31.649727    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fd92500_8430_4f72_8585_0d3d82cdf42c.slice/crio-c76c50bd3ea892f22bb1d1e192933919e9db44447fbfe08de946046db72a3d03 WatchSource:0}: Error finding container c76c50bd3ea892f22bb1d1e192933919e9db44447fbfe08de946046db72a3d03: Status 404 returned error can't find the container with id c76c50bd3ea892f22bb1d1e192933919e9db44447fbfe08de946046db72a3d03
Sep 15 20:54:31.834980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.834937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" event={"ID":"9fd92500-8430-4f72-8585-0d3d82cdf42c","Type":"ContainerStarted","Data":"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"}
Sep 15 20:54:31.835176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.834997    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" event={"ID":"9fd92500-8430-4f72-8585-0d3d82cdf42c","Type":"ContainerStarted","Data":"c76c50bd3ea892f22bb1d1e192933919e9db44447fbfe08de946046db72a3d03"}
Sep 15 20:54:31.835176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.835125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:31.850988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:31.850941    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" podStartSLOduration=0.850924731 podStartE2EDuration="850.924731ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:54:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:54:31.848816611 +0000 UTC m=+1062.635940641" watchObservedRunningTime="2026-09-15 20:54:31.850924731 +0000 UTC m=+1062.638048760"
Sep 15 20:54:35.374592 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:35.374562    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:54:35.797803 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:35.797722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:54:37.812101 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:37.812073    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:42.844142 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:42.844065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 20:54:47.896417 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:47.896385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:47.897049 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:47.896604    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" podUID="86a2a161-b0d8-4555-81bf-8121cea63849" containerName="payload-pre-processing" containerID="cri-o://1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e" gracePeriod=30
Sep 15 20:54:48.161348 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.161280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:48.296704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.296675    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume\") pod \"86a2a161-b0d8-4555-81bf-8121cea63849\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") "
Sep 15 20:54:48.296899 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.296756    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l77k\" (UniqueName: \"kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k\") pod \"86a2a161-b0d8-4555-81bf-8121cea63849\" (UID: \"86a2a161-b0d8-4555-81bf-8121cea63849\") "
Sep 15 20:54:48.297103 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.297076    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86a2a161-b0d8-4555-81bf-8121cea63849" (UID: "86a2a161-b0d8-4555-81bf-8121cea63849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:54:48.298958 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.298928    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k" (OuterVolumeSpecName: "kube-api-access-2l77k") pod "86a2a161-b0d8-4555-81bf-8121cea63849" (UID: "86a2a161-b0d8-4555-81bf-8121cea63849"). InnerVolumeSpecName "kube-api-access-2l77k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:54:48.398504 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.398467    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86a2a161-b0d8-4555-81bf-8121cea63849-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:54:48.398504 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.398504    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l77k\" (UniqueName: \"kubernetes.io/projected/86a2a161-b0d8-4555-81bf-8121cea63849-kube-api-access-2l77k\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:54:48.922988 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.922959    2579 generic.go:356] "Generic (PLEG): container finished" podID="86a2a161-b0d8-4555-81bf-8121cea63849" containerID="1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e" exitCode=0
Sep 15 20:54:48.923510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.923003    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"
Sep 15 20:54:48.923510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.923059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" event={"ID":"86a2a161-b0d8-4555-81bf-8121cea63849","Type":"ContainerDied","Data":"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"}
Sep 15 20:54:48.923510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.923122    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj" event={"ID":"86a2a161-b0d8-4555-81bf-8121cea63849","Type":"ContainerDied","Data":"51ce60a640a9d2d1826c952275d33a2cc60cd1c84d95535b620341035e6c6b11"}
Sep 15 20:54:48.923510 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.923153    2579 scope.go:117] "RemoveContainer" containerID="1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"
Sep 15 20:54:48.936221 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.936202    2579 scope.go:117] "RemoveContainer" containerID="1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"
Sep 15 20:54:48.936498 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:54:48.936480    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e\": container with ID starting with 1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e not found: ID does not exist" containerID="1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"
Sep 15 20:54:48.936555 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.936505    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e"} err="failed to get container status \"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e\": rpc error: code = NotFound desc = could not find container \"1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e\": container with ID starting with 1bb0450a4729be31355d67f00aa9ccc4881eff284b5eea72692e905c5ae7bd9e not found: ID does not exist"
Sep 15 20:54:48.949994 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.949972    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:48.954173 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:48.954156    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ded67cd1-7984b56f66-z49xj"]
Sep 15 20:54:49.671090 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:54:49.671053    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a2a161-b0d8-4555-81bf-8121cea63849" path="/var/lib/kubelet/pods/86a2a161-b0d8-4555-81bf-8121cea63849/volumes"
Sep 15 20:55:03.279574 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:03.279539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:55:17.564298 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:17.564250    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:55:29.810106 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.810065    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:55:29.810765 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.810746    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a2a161-b0d8-4555-81bf-8121cea63849" containerName="payload-pre-processing"
Sep 15 20:55:29.810844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.810769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a2a161-b0d8-4555-81bf-8121cea63849" containerName="payload-pre-processing"
Sep 15 20:55:29.810900 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.810876    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a2a161-b0d8-4555-81bf-8121cea63849" containerName="payload-pre-processing"
Sep 15 20:55:29.814256 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.814236    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.816696 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.816675    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0143258d-dockercfg-2jc95\""
Sep 15 20:55:29.816919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.816907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0143258d\""
Sep 15 20:55:29.823321 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.823293    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:55:29.876929 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.876878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47jj\" (UniqueName: \"kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.877122 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.876999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.977574 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.977538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.977769 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.977614    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47jj\" (UniqueName: \"kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.978322 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.978300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:29.986567 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:29.986545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47jj\" (UniqueName: \"kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj\") pod \"payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:30.125621 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:30.125592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:30.397291 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:30.397266    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:55:30.399495 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:55:30.399467    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2491c5dc_1a55_4be7_9bfb_b548010b181b.slice/crio-5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a WatchSource:0}: Error finding container 5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a: Status 404 returned error can't find the container with id 5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a
Sep 15 20:55:31.135854 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:31.135819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" event={"ID":"2491c5dc-1a55-4be7-9bfb-b548010b181b","Type":"ContainerStarted","Data":"77138d50c14f269b2a2f0f1e9ba4bb9ef0ab2ede34aedb7447db4b7578bf4cbe"}
Sep 15 20:55:31.136427 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:31.135882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" event={"ID":"2491c5dc-1a55-4be7-9bfb-b548010b181b","Type":"ContainerStarted","Data":"5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a"}
Sep 15 20:55:31.136427 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:31.135915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:31.155552 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:31.155498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" podStartSLOduration=2.155483229 podStartE2EDuration="2.155483229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:31.154049096 +0000 UTC m=+1121.941173126" watchObservedRunningTime="2026-09-15 20:55:31.155483229 +0000 UTC m=+1121.942607258"
Sep 15 20:55:33.067821 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:33.067774    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:55:42.143199 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:42.143166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:54.164305 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.164270    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 20:55:54.167674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.167655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.171505 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.171487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"openshift-service-ca.crt\""
Sep 15 20:55:54.171505 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.171497    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"kube-root-ca.crt\""
Sep 15 20:55:54.171653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.171499    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"simulator-w0-3f1c93-kserve-self-signed-certs\""
Sep 15 20:55:54.171653 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.171492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"default-dockercfg-qswjz\""
Sep 15 20:55:54.177230 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.177211    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 20:55:54.305041 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.304983    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.305223 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.305053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.305223 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.305090    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.305223 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.305126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.305223 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.305206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.305425 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.305305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fv8w\" (UniqueName: \"kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.405774 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.405744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fv8w\" (UniqueName: \"kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.405940 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.405795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.405940 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.405909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.405964    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.405997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406053 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.406050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406326 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.406303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406432 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.406345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.406432 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.406402    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.408322 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.408304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.408656 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.408637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.414371 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.414327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fv8w\" (UniqueName: \"kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w\") pod \"simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") " pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.478001 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.477961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:55:54.756440 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:54.756416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 20:55:54.758225 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:55:54.758192    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c84908c_bdf9_4ef6_9e28_5f2f8ce15761.slice/crio-a2ea87b6f9967d10b2a6bf4b225f81ac446ae024b453a6f6683b5a3b114216d6 WatchSource:0}: Error finding container a2ea87b6f9967d10b2a6bf4b225f81ac446ae024b453a6f6683b5a3b114216d6: Status 404 returned error can't find the container with id a2ea87b6f9967d10b2a6bf4b225f81ac446ae024b453a6f6683b5a3b114216d6
Sep 15 20:55:55.267904 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:55.267869    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" event={"ID":"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761","Type":"ContainerStarted","Data":"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"}
Sep 15 20:55:55.267904 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:55.267910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" event={"ID":"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761","Type":"ContainerStarted","Data":"a2ea87b6f9967d10b2a6bf4b225f81ac446ae024b453a6f6683b5a3b114216d6"}
Sep 15 20:55:55.287592 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:55.287545    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" podStartSLOduration=1.287530451 podStartE2EDuration="1.287530451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:55:55.286647663 +0000 UTC m=+1146.073771886" watchObservedRunningTime="2026-09-15 20:55:55.287530451 +0000 UTC m=+1146.074654480"
Sep 15 20:55:59.032825 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.032790    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:55:59.033288 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.033056    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" podUID="2491c5dc-1a55-4be7-9bfb-b548010b181b" containerName="payload-pre-processing" containerID="cri-o://77138d50c14f269b2a2f0f1e9ba4bb9ef0ab2ede34aedb7447db4b7578bf4cbe" gracePeriod=30
Sep 15 20:55:59.294590 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.294485    2579 generic.go:356] "Generic (PLEG): container finished" podID="2491c5dc-1a55-4be7-9bfb-b548010b181b" containerID="77138d50c14f269b2a2f0f1e9ba4bb9ef0ab2ede34aedb7447db4b7578bf4cbe" exitCode=0
Sep 15 20:55:59.294590 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.294565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" event={"ID":"2491c5dc-1a55-4be7-9bfb-b548010b181b","Type":"ContainerDied","Data":"77138d50c14f269b2a2f0f1e9ba4bb9ef0ab2ede34aedb7447db4b7578bf4cbe"}
Sep 15 20:55:59.294817 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.294627    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg" event={"ID":"2491c5dc-1a55-4be7-9bfb-b548010b181b","Type":"ContainerDied","Data":"5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a"}
Sep 15 20:55:59.294817 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.294650    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ee570c2e3ec1a6dd2797f279e4a22e36edd9aa0bca9b42d7542086c7cd41c5a"
Sep 15 20:55:59.306938 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.306910    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:55:59.455541 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.455495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume\") pod \"2491c5dc-1a55-4be7-9bfb-b548010b181b\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") "
Sep 15 20:55:59.455752 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.455640    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d47jj\" (UniqueName: \"kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj\") pod \"2491c5dc-1a55-4be7-9bfb-b548010b181b\" (UID: \"2491c5dc-1a55-4be7-9bfb-b548010b181b\") "
Sep 15 20:55:59.455896 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.455874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2491c5dc-1a55-4be7-9bfb-b548010b181b" (UID: "2491c5dc-1a55-4be7-9bfb-b548010b181b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:55:59.457929 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.457898    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj" (OuterVolumeSpecName: "kube-api-access-d47jj") pod "2491c5dc-1a55-4be7-9bfb-b548010b181b" (UID: "2491c5dc-1a55-4be7-9bfb-b548010b181b"). InnerVolumeSpecName "kube-api-access-d47jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:55:59.556741 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.556653    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47jj\" (UniqueName: \"kubernetes.io/projected/2491c5dc-1a55-4be7-9bfb-b548010b181b-kube-api-access-d47jj\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:55:59.556741 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:55:59.556685    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2491c5dc-1a55-4be7-9bfb-b548010b181b-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:56:00.299873 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:00.299839    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"
Sep 15 20:56:00.318065 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:00.318035    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:56:00.324181 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:00.324149    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0143258d-5f88fbd5b6-p2pkg"]
Sep 15 20:56:00.677561 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:00.677521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:56:01.669320 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:01.669285    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2491c5dc-1a55-4be7-9bfb-b548010b181b" path="/var/lib/kubelet/pods/2491c5dc-1a55-4be7-9bfb-b548010b181b/volumes"
Sep 15 20:56:04.478463 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:04.478422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:56:04.478958 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:04.478477    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:56:04.491549 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:04.491515    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:56:05.339079 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:05.339048    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 20:56:09.095390 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.095358    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:09.095798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.095755    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2491c5dc-1a55-4be7-9bfb-b548010b181b" containerName="payload-pre-processing"
Sep 15 20:56:09.095798 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.095766    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2491c5dc-1a55-4be7-9bfb-b548010b181b" containerName="payload-pre-processing"
Sep 15 20:56:09.095878 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.095823    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2491c5dc-1a55-4be7-9bfb-b548010b181b" containerName="payload-pre-processing"
Sep 15 20:56:09.100176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.100158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.102743 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.102718    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a7948686\""
Sep 15 20:56:09.102875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.102835    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a7948686-dockercfg-2tvtv\""
Sep 15 20:56:09.108930 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.108906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:09.148560 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.148529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.148704 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.148598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mll8\" (UniqueName: \"kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.249833 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.249801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.250044 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.249862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mll8\" (UniqueName: \"kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.250570 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.250552    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.257919 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.257900    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mll8\" (UniqueName: \"kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8\") pod \"payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.411095 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.411067    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:09.687757 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:09.687695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:09.690190 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:56:09.690162    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60acc8a5_97d8_4da9_b890_ac9538b98d42.slice/crio-ebdeaee10448adbc47efeaa1f72d7e3621b4ef63c66f2497c956a9b35256d2ba WatchSource:0}: Error finding container ebdeaee10448adbc47efeaa1f72d7e3621b4ef63c66f2497c956a9b35256d2ba: Status 404 returned error can't find the container with id ebdeaee10448adbc47efeaa1f72d7e3621b4ef63c66f2497c956a9b35256d2ba
Sep 15 20:56:10.354802 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:10.354761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" event={"ID":"60acc8a5-97d8-4da9-b890-ac9538b98d42","Type":"ContainerStarted","Data":"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"}
Sep 15 20:56:10.355245 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:10.354827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" event={"ID":"60acc8a5-97d8-4da9-b890-ac9538b98d42","Type":"ContainerStarted","Data":"ebdeaee10448adbc47efeaa1f72d7e3621b4ef63c66f2497c956a9b35256d2ba"}
Sep 15 20:56:10.355245 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:10.354954    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:10.372898 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:10.372844    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" podStartSLOduration=1.37282705 podStartE2EDuration="1.37282705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:10.371562718 +0000 UTC m=+1161.158686760" watchObservedRunningTime="2026-09-15 20:56:10.37282705 +0000 UTC m=+1161.159951079"
Sep 15 20:56:13.529846 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.529751    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:56:13.530301 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.530033    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" podUID="e29d509d-c73a-42ae-8947-8696b9ac7d85" containerName="payload-processing" containerID="cri-o://ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5" gracePeriod=30
Sep 15 20:56:13.802462 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.802373    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:56:13.898613 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.898585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume\") pod \"e29d509d-c73a-42ae-8947-8696b9ac7d85\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") "
Sep 15 20:56:13.898805 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.898649    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8n5z\" (UniqueName: \"kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z\") pod \"e29d509d-c73a-42ae-8947-8696b9ac7d85\" (UID: \"e29d509d-c73a-42ae-8947-8696b9ac7d85\") "
Sep 15 20:56:13.899052 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.898996    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e29d509d-c73a-42ae-8947-8696b9ac7d85" (UID: "e29d509d-c73a-42ae-8947-8696b9ac7d85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:56:13.900898 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.900877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z" (OuterVolumeSpecName: "kube-api-access-h8n5z") pod "e29d509d-c73a-42ae-8947-8696b9ac7d85" (UID: "e29d509d-c73a-42ae-8947-8696b9ac7d85"). InnerVolumeSpecName "kube-api-access-h8n5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:56:13.999964 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.999933    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e29d509d-c73a-42ae-8947-8696b9ac7d85-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:56:13.999964 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:13.999958    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8n5z\" (UniqueName: \"kubernetes.io/projected/e29d509d-c73a-42ae-8947-8696b9ac7d85-kube-api-access-h8n5z\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:56:14.375133 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.375095    2579 generic.go:356] "Generic (PLEG): container finished" podID="e29d509d-c73a-42ae-8947-8696b9ac7d85" containerID="ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5" exitCode=0
Sep 15 20:56:14.375304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.375144    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"
Sep 15 20:56:14.375304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.375185    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" event={"ID":"e29d509d-c73a-42ae-8947-8696b9ac7d85","Type":"ContainerDied","Data":"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"}
Sep 15 20:56:14.375304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.375238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw" event={"ID":"e29d509d-c73a-42ae-8947-8696b9ac7d85","Type":"ContainerDied","Data":"142ae168489cd54909ab773b9ce516068cf41c0ffd284ed608910ada218ba1f3"}
Sep 15 20:56:14.375304 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.375260    2579 scope.go:117] "RemoveContainer" containerID="ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"
Sep 15 20:56:14.384585 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.384561    2579 scope.go:117] "RemoveContainer" containerID="ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"
Sep 15 20:56:14.384935 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:56:14.384910    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5\": container with ID starting with ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5 not found: ID does not exist" containerID="ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"
Sep 15 20:56:14.385050 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.384945    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5"} err="failed to get container status \"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5\": rpc error: code = NotFound desc = could not find container \"ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5\": container with ID starting with ba7ce06e8c25eab10d40f9f363914a6614fdc17333f462f07aa688fd7ac2fea5 not found: ID does not exist"
Sep 15 20:56:14.402132 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.402107    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:56:14.407672 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:14.407649    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b9ff68-c4fc7d979-dk8gw"]
Sep 15 20:56:15.668836 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:15.668807    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29d509d-c73a-42ae-8947-8696b9ac7d85" path="/var/lib/kubelet/pods/e29d509d-c73a-42ae-8947-8696b9ac7d85/volumes"
Sep 15 20:56:19.183285 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:19.183247    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:56:21.361866 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:21.361839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:25.850986 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.850949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 20:56:25.851456 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.851404    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29d509d-c73a-42ae-8947-8696b9ac7d85" containerName="payload-processing"
Sep 15 20:56:25.851456 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.851417    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29d509d-c73a-42ae-8947-8696b9ac7d85" containerName="payload-processing"
Sep 15 20:56:25.851531 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.851484    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29d509d-c73a-42ae-8947-8696b9ac7d85" containerName="payload-processing"
Sep 15 20:56:25.854656 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.854636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:25.858738 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.858717    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5b5b9\"/\"default-dockercfg-b5v5v\""
Sep 15 20:56:25.858844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.858736    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d5b5b9\"/\"test-model-d5b5b9-kserve-self-signed-certs\""
Sep 15 20:56:25.858844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.858746    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5b5b9\"/\"kube-root-ca.crt\""
Sep 15 20:56:25.858844 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.858738    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d5b5b9\"/\"openshift-service-ca.crt\""
Sep 15 20:56:25.864515 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:25.864486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 20:56:26.010041 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.009967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.010041 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.010038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h6jr\" (UniqueName: \"kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.010263 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.010075    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.010263 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.010113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.010263 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.010172    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.010263 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.010240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111603    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111649    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h6jr\" (UniqueName: \"kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111707 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.111993 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.111845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.112201 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.112050    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.112201 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.112182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.112312 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.112208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.114282 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.114252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.114469 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.114453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.120209 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.120189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h6jr\" (UniqueName: \"kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr\") pod \"test-model-d5b5b9-kserve-b8ccfbd7-7mp9r\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") " pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.165186 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.165152    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:26.438176 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:26.438150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 20:56:26.440295 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:56:26.440268    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e94a744_b422_4c3d_bf4f_be619367f375.slice/crio-3ac10d8f3efbea1b53e338171319937627c8ff698a4e3ed990cfda4a28670e06 WatchSource:0}: Error finding container 3ac10d8f3efbea1b53e338171319937627c8ff698a4e3ed990cfda4a28670e06: Status 404 returned error can't find the container with id 3ac10d8f3efbea1b53e338171319937627c8ff698a4e3ed990cfda4a28670e06
Sep 15 20:56:27.448997 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:27.448966    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" event={"ID":"4e94a744-b422-4c3d-bf4f-be619367f375","Type":"ContainerStarted","Data":"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"}
Sep 15 20:56:27.449378 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:27.449030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" event={"ID":"4e94a744-b422-4c3d-bf4f-be619367f375","Type":"ContainerStarted","Data":"3ac10d8f3efbea1b53e338171319937627c8ff698a4e3ed990cfda4a28670e06"}
Sep 15 20:56:27.467749 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:27.467707    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" podStartSLOduration=2.46769192 podStartE2EDuration="2.46769192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:56:27.465878406 +0000 UTC m=+1178.253002434" watchObservedRunningTime="2026-09-15 20:56:27.46769192 +0000 UTC m=+1178.254815949"
Sep 15 20:56:29.163291 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.163258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:29.163761 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.163513    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" podUID="60acc8a5-97d8-4da9-b890-ac9538b98d42" containerName="payload-pre-processing" containerID="cri-o://3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5" gracePeriod=30
Sep 15 20:56:29.425335 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.425268    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:29.464875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.464652    2579 generic.go:356] "Generic (PLEG): container finished" podID="60acc8a5-97d8-4da9-b890-ac9538b98d42" containerID="3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5" exitCode=0
Sep 15 20:56:29.464875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.464721    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"
Sep 15 20:56:29.464875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.464748    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" event={"ID":"60acc8a5-97d8-4da9-b890-ac9538b98d42","Type":"ContainerDied","Data":"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"}
Sep 15 20:56:29.464875 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.464803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k" event={"ID":"60acc8a5-97d8-4da9-b890-ac9538b98d42","Type":"ContainerDied","Data":"ebdeaee10448adbc47efeaa1f72d7e3621b4ef63c66f2497c956a9b35256d2ba"}
Sep 15 20:56:29.465291 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.465154    2579 scope.go:117] "RemoveContainer" containerID="3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"
Sep 15 20:56:29.480108 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.480087    2579 scope.go:117] "RemoveContainer" containerID="3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"
Sep 15 20:56:29.480658 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:56:29.480614    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5\": container with ID starting with 3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5 not found: ID does not exist" containerID="3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"
Sep 15 20:56:29.480799 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.480657    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5"} err="failed to get container status \"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5\": rpc error: code = NotFound desc = could not find container \"3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5\": container with ID starting with 3351485a6f97809f7300bbcc4d1990ea7d9e2cad0656ffd49c8dfe451560e1e5 not found: ID does not exist"
Sep 15 20:56:29.549839 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.549797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mll8\" (UniqueName: \"kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8\") pod \"60acc8a5-97d8-4da9-b890-ac9538b98d42\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") "
Sep 15 20:56:29.550042 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.549884    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume\") pod \"60acc8a5-97d8-4da9-b890-ac9538b98d42\" (UID: \"60acc8a5-97d8-4da9-b890-ac9538b98d42\") "
Sep 15 20:56:29.550290 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.550264    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60acc8a5-97d8-4da9-b890-ac9538b98d42" (UID: "60acc8a5-97d8-4da9-b890-ac9538b98d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:56:29.552095 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.552057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8" (OuterVolumeSpecName: "kube-api-access-5mll8") pod "60acc8a5-97d8-4da9-b890-ac9538b98d42" (UID: "60acc8a5-97d8-4da9-b890-ac9538b98d42"). InnerVolumeSpecName "kube-api-access-5mll8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:56:29.651100 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.651061    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mll8\" (UniqueName: \"kubernetes.io/projected/60acc8a5-97d8-4da9-b890-ac9538b98d42-kube-api-access-5mll8\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:56:29.651100 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.651099    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60acc8a5-97d8-4da9-b890-ac9538b98d42-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:56:29.780629 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.780592    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:29.783505 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:29.783479    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a7948686-c54fc769d-7lh6k"]
Sep 15 20:56:31.668700 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:31.668663    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60acc8a5-97d8-4da9-b890-ac9538b98d42" path="/var/lib/kubelet/pods/60acc8a5-97d8-4da9-b890-ac9538b98d42/volumes"
Sep 15 20:56:36.165605 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:36.165574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:36.166060 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:36.165632    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:36.177851 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:36.177828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:36.519910 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:36.519835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 20:56:38.828068 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:56:38.828034    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:57:16.477904 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.477870    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:57:16.478350 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.478288    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60acc8a5-97d8-4da9-b890-ac9538b98d42" containerName="payload-pre-processing"
Sep 15 20:57:16.478350 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.478299    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="60acc8a5-97d8-4da9-b890-ac9538b98d42" containerName="payload-pre-processing"
Sep 15 20:57:16.478422 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.478365    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="60acc8a5-97d8-4da9-b890-ac9538b98d42" containerName="payload-pre-processing"
Sep 15 20:57:16.481356 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.481339    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.486126 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.486105    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-41588d\""
Sep 15 20:57:16.488166 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.488144    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-41588d-dockercfg-57rk4\""
Sep 15 20:57:16.496853 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.496832    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:57:16.603818 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.603787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6s8d\" (UniqueName: \"kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.604003 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.603855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.704402 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.704365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6s8d\" (UniqueName: \"kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.704598 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.704436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.705209 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.705187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.713632 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.713608    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6s8d\" (UniqueName: \"kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d\") pod \"payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") " pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:16.791211 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:16.791124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:17.068514 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:17.068492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:57:17.071642 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:57:17.071612    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe480aa_58bd_4003_addf_492511fdae2b.slice/crio-98d835b4dedaa43988f1f7e71dbcd20a0bda3fcedfeb451d2985687785f58a17 WatchSource:0}: Error finding container 98d835b4dedaa43988f1f7e71dbcd20a0bda3fcedfeb451d2985687785f58a17: Status 404 returned error can't find the container with id 98d835b4dedaa43988f1f7e71dbcd20a0bda3fcedfeb451d2985687785f58a17
Sep 15 20:57:17.714826 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:17.714794    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" event={"ID":"6fe480aa-58bd-4003-addf-492511fdae2b","Type":"ContainerStarted","Data":"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"}
Sep 15 20:57:17.715248 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:17.714850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" event={"ID":"6fe480aa-58bd-4003-addf-492511fdae2b","Type":"ContainerStarted","Data":"98d835b4dedaa43988f1f7e71dbcd20a0bda3fcedfeb451d2985687785f58a17"}
Sep 15 20:57:17.715248 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:17.714878    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:17.733122 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:17.733073    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" podStartSLOduration=1.733056989 podStartE2EDuration="1.733056989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:57:17.731091452 +0000 UTC m=+1228.518215482" watchObservedRunningTime="2026-09-15 20:57:17.733056989 +0000 UTC m=+1228.520181018"
Sep 15 20:57:25.114338 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:25.114309    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:57:28.721971 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:28.721942    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:57:41.476530 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:41.476443    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:57:55.575767 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:57:55.575730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:58:19.540058 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.540024    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:58:19.543965 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.543943    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.547289 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.547267    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c103b8e8-dockercfg-k8x59\""
Sep 15 20:58:19.547403 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.547311    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c103b8e8\""
Sep 15 20:58:19.554931 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.554905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:58:19.597306 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.597274    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9g4q\" (UniqueName: \"kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.597486 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.597327    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.698229 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.698190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9g4q\" (UniqueName: \"kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.698439 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.698274    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.699101 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.699068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.707151 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.707120    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9g4q\" (UniqueName: \"kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q\") pod \"payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") " pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:19.855951 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:19.855860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:20.154437 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:20.154412    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:58:20.156609 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:58:20.156583    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfd2cf4_322f_4034_a31d_e9f6f1aa802e.slice/crio-97b366b274e3168d65acc67ae4f07db98d54aec798d8fa52cc1b6226ac04f7d7 WatchSource:0}: Error finding container 97b366b274e3168d65acc67ae4f07db98d54aec798d8fa52cc1b6226ac04f7d7: Status 404 returned error can't find the container with id 97b366b274e3168d65acc67ae4f07db98d54aec798d8fa52cc1b6226ac04f7d7
Sep 15 20:58:21.043445 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:21.043404    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" event={"ID":"edfd2cf4-322f-4034-a31d-e9f6f1aa802e","Type":"ContainerStarted","Data":"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"}
Sep 15 20:58:21.043839 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:21.043501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" event={"ID":"edfd2cf4-322f-4034-a31d-e9f6f1aa802e","Type":"ContainerStarted","Data":"97b366b274e3168d65acc67ae4f07db98d54aec798d8fa52cc1b6226ac04f7d7"}
Sep 15 20:58:21.043839 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:21.043529    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:21.061024 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:21.060963    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" podStartSLOduration=2.060947664 podStartE2EDuration="2.060947664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:58:21.059831994 +0000 UTC m=+1291.846956027" watchObservedRunningTime="2026-09-15 20:58:21.060947664 +0000 UTC m=+1291.848071693"
Sep 15 20:58:21.271797 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:21.271758    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:58:32.051361 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:32.051332    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:58:43.867498 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:58:43.867463    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:59:03.405387 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.405357    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:59:03.405916 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.405563    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" podUID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" containerName="payload-processing" containerID="cri-o://2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f" gracePeriod=30
Sep 15 20:59:03.677944 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.677915    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:59:03.703604 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.703573    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9g4q\" (UniqueName: \"kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q\") pod \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") "
Sep 15 20:59:03.703792 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.703614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume\") pod \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\" (UID: \"edfd2cf4-322f-4034-a31d-e9f6f1aa802e\") "
Sep 15 20:59:03.704081 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.704059    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edfd2cf4-322f-4034-a31d-e9f6f1aa802e" (UID: "edfd2cf4-322f-4034-a31d-e9f6f1aa802e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:59:03.705781 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.705758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q" (OuterVolumeSpecName: "kube-api-access-x9g4q") pod "edfd2cf4-322f-4034-a31d-e9f6f1aa802e" (UID: "edfd2cf4-322f-4034-a31d-e9f6f1aa802e"). InnerVolumeSpecName "kube-api-access-x9g4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:59:03.804382 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.804350    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9g4q\" (UniqueName: \"kubernetes.io/projected/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-kube-api-access-x9g4q\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:59:03.804382 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:03.804375    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edfd2cf4-322f-4034-a31d-e9f6f1aa802e-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:59:04.266745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.266709    2579 generic.go:356] "Generic (PLEG): container finished" podID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" containerID="2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f" exitCode=0
Sep 15 20:59:04.266745 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.266749    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" event={"ID":"edfd2cf4-322f-4034-a31d-e9f6f1aa802e","Type":"ContainerDied","Data":"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"}
Sep 15 20:59:04.266980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.266754    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"
Sep 15 20:59:04.266980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.266800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj" event={"ID":"edfd2cf4-322f-4034-a31d-e9f6f1aa802e","Type":"ContainerDied","Data":"97b366b274e3168d65acc67ae4f07db98d54aec798d8fa52cc1b6226ac04f7d7"}
Sep 15 20:59:04.266980 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.266827    2579 scope.go:117] "RemoveContainer" containerID="2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"
Sep 15 20:59:04.282859 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.282841    2579 scope.go:117] "RemoveContainer" containerID="2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"
Sep 15 20:59:04.283373 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:59:04.283161    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f\": container with ID starting with 2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f not found: ID does not exist" containerID="2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"
Sep 15 20:59:04.283373 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.283190    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f"} err="failed to get container status \"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f\": rpc error: code = NotFound desc = could not find container \"2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f\": container with ID starting with 2d0be2d2769672969520548128f0fbc49afed8c683db64e2dd24396b28e49a8f not found: ID does not exist"
Sep 15 20:59:04.297743 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.297715    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:59:04.303350 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:04.303324    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c103b8e8-7d94b7d7fc-45thj"]
Sep 15 20:59:05.495595 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:05.495556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:59:05.668535 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:05.668501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" path="/var/lib/kubelet/pods/edfd2cf4-322f-4034-a31d-e9f6f1aa802e/volumes"
Sep 15 20:59:29.735855 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:29.735814    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:59:29.736352 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:29.736107    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" podUID="6fe480aa-58bd-4003-addf-492511fdae2b" containerName="payload-processing" containerID="cri-o://40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9" gracePeriod=30
Sep 15 20:59:30.019870 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.019847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:59:30.155302 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.155272    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6s8d\" (UniqueName: \"kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d\") pod \"6fe480aa-58bd-4003-addf-492511fdae2b\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") "
Sep 15 20:59:30.155489 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.155435    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume\") pod \"6fe480aa-58bd-4003-addf-492511fdae2b\" (UID: \"6fe480aa-58bd-4003-addf-492511fdae2b\") "
Sep 15 20:59:30.155828 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.155804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6fe480aa-58bd-4003-addf-492511fdae2b" (UID: "6fe480aa-58bd-4003-addf-492511fdae2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 20:59:30.157674 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.157639    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d" (OuterVolumeSpecName: "kube-api-access-k6s8d") pod "6fe480aa-58bd-4003-addf-492511fdae2b" (UID: "6fe480aa-58bd-4003-addf-492511fdae2b"). InnerVolumeSpecName "kube-api-access-k6s8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 20:59:30.256546 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.256470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6s8d\" (UniqueName: \"kubernetes.io/projected/6fe480aa-58bd-4003-addf-492511fdae2b-kube-api-access-k6s8d\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:59:30.256546 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.256497    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fe480aa-58bd-4003-addf-492511fdae2b-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 20:59:30.403647 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.403614    2579 generic.go:356] "Generic (PLEG): container finished" podID="6fe480aa-58bd-4003-addf-492511fdae2b" containerID="40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9" exitCode=0
Sep 15 20:59:30.403841 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.403656    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"
Sep 15 20:59:30.403841 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.403701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" event={"ID":"6fe480aa-58bd-4003-addf-492511fdae2b","Type":"ContainerDied","Data":"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"}
Sep 15 20:59:30.403841 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.403778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46" event={"ID":"6fe480aa-58bd-4003-addf-492511fdae2b","Type":"ContainerDied","Data":"98d835b4dedaa43988f1f7e71dbcd20a0bda3fcedfeb451d2985687785f58a17"}
Sep 15 20:59:30.403841 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.403803    2579 scope.go:117] "RemoveContainer" containerID="40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"
Sep 15 20:59:30.413165 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.413149    2579 scope.go:117] "RemoveContainer" containerID="40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"
Sep 15 20:59:30.413418 ip-10-0-138-122 kubenswrapper[2579]: E0915 20:59:30.413399    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9\": container with ID starting with 40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9 not found: ID does not exist" containerID="40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"
Sep 15 20:59:30.413482 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.413429    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9"} err="failed to get container status \"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9\": rpc error: code = NotFound desc = could not find container \"40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9\": container with ID starting with 40ab07b7c90ffdd5a557f36dd15a6f5eda5c2f076487cad0cb808fa07808dfd9 not found: ID does not exist"
Sep 15 20:59:30.426978 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.426955    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:59:30.430987 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:30.430962    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-41588d-648cdbcc97-bqk46"]
Sep 15 20:59:31.323783 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.323736    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 20:59:31.324195 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324177    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" containerName="payload-processing"
Sep 15 20:59:31.324195 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324190    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" containerName="payload-processing"
Sep 15 20:59:31.324270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324201    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe480aa-58bd-4003-addf-492511fdae2b" containerName="payload-processing"
Sep 15 20:59:31.324270 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324207    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe480aa-58bd-4003-addf-492511fdae2b" containerName="payload-processing"
Sep 15 20:59:31.324534 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324277    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe480aa-58bd-4003-addf-492511fdae2b" containerName="payload-processing"
Sep 15 20:59:31.324534 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.324288    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="edfd2cf4-322f-4034-a31d-e9f6f1aa802e" containerName="payload-processing"
Sep 15 20:59:31.328972 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.328951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.332149 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.332129    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-de4215\""
Sep 15 20:59:31.346096 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.346074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 20:59:31.381141 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.381115    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 20:59:31.384625 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.384604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.387403 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.387387    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-de4215-dockercfg-js69k\""
Sep 15 20:59:31.397700 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.397680    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 20:59:31.469281 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.469213    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.469281 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.469265    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj9tb\" (UniqueName: \"kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.570673 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.570643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.570673 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.570678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj9tb\" (UniqueName: \"kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.570898 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.570753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.570898 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.570782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvvw\" (UniqueName: \"kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.571471 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.571453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.579294 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.579217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj9tb\" (UniqueName: \"kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb\") pod \"payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.638679 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.638641    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:31.670120 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.670086    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe480aa-58bd-4003-addf-492511fdae2b" path="/var/lib/kubelet/pods/6fe480aa-58bd-4003-addf-492511fdae2b/volumes"
Sep 15 20:59:31.671550 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.671525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.671670 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.671579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvvw\" (UniqueName: \"kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.672295 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.672279    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.683416 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.683385    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvvw\" (UniqueName: \"kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw\") pod \"payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.695391 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.695369    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:31.915367 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.915344    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 20:59:31.917438 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:59:31.917412    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8a07c2_4b70_4f15_8936_2642627d8363.slice/crio-d074f4f6f6e55f3ce3d10028a4985a5b85e17488674753186f72edaad600ec64 WatchSource:0}: Error finding container d074f4f6f6e55f3ce3d10028a4985a5b85e17488674753186f72edaad600ec64: Status 404 returned error can't find the container with id d074f4f6f6e55f3ce3d10028a4985a5b85e17488674753186f72edaad600ec64
Sep 15 20:59:31.981462 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:31.981437    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 20:59:31.983319 ip-10-0-138-122 kubenswrapper[2579]: W0915 20:59:31.983295    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59e13b92_3837_4acd_a5ec_e4edf0776714.slice/crio-e051d781b2e4be032525e6b686706ad9a1a92813ca7739b59247d3977af70880 WatchSource:0}: Error finding container e051d781b2e4be032525e6b686706ad9a1a92813ca7739b59247d3977af70880: Status 404 returned error can't find the container with id e051d781b2e4be032525e6b686706ad9a1a92813ca7739b59247d3977af70880
Sep 15 20:59:32.418868 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.418833    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" event={"ID":"59e13b92-3837-4acd-a5ec-e4edf0776714","Type":"ContainerStarted","Data":"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"}
Sep 15 20:59:32.419390 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.418894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" event={"ID":"59e13b92-3837-4acd-a5ec-e4edf0776714","Type":"ContainerStarted","Data":"e051d781b2e4be032525e6b686706ad9a1a92813ca7739b59247d3977af70880"}
Sep 15 20:59:32.419390 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.418927    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:32.420591 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.420564    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" event={"ID":"cb8a07c2-4b70-4f15-8936-2642627d8363","Type":"ContainerStarted","Data":"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"}
Sep 15 20:59:32.420714 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.420611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" event={"ID":"cb8a07c2-4b70-4f15-8936-2642627d8363","Type":"ContainerStarted","Data":"d074f4f6f6e55f3ce3d10028a4985a5b85e17488674753186f72edaad600ec64"}
Sep 15 20:59:32.420714 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.420638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 20:59:32.436994 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.436952    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" podStartSLOduration=1.4369380760000001 podStartE2EDuration="1.436938076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:32.434567504 +0000 UTC m=+1363.221691619" watchObservedRunningTime="2026-09-15 20:59:32.436938076 +0000 UTC m=+1363.224062105"
Sep 15 20:59:32.451838 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:32.451786    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" podStartSLOduration=1.451770532 podStartE2EDuration="1.451770532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 20:59:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 20:59:32.449843476 +0000 UTC m=+1363.236967504" watchObservedRunningTime="2026-09-15 20:59:32.451770532 +0000 UTC m=+1363.238894560"
Sep 15 20:59:36.276131 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:36.276101    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 20:59:43.428211 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:43.428176    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 20:59:43.428590 ip-10-0-138-122 kubenswrapper[2579]: I0915 20:59:43.428351    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 21:00:00.159049 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.158997    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"]
Sep 15 21:00:00.164872 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.164849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"
Sep 15 21:00:00.167586 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.167552    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\""
Sep 15 21:00:00.168081 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.168055    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"]
Sep 15 21:00:00.178825 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.178801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"
Sep 15 21:00:00.457899 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.457872    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"]
Sep 15 21:00:00.459998 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:00:00.459974    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c15548_5553_4c41_8b5d_5818e5dedd04.slice/crio-2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1 WatchSource:0}: Error finding container 2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1: Status 404 returned error can't find the container with id 2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1
Sep 15 21:00:00.461937 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.461923    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:00:00.575141 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:00.575104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9" event={"ID":"05c15548-5553-4c41-8b5d-5818e5dedd04","Type":"ContainerStarted","Data":"2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1"}
Sep 15 21:00:03.592538 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:03.592440    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9" event={"ID":"05c15548-5553-4c41-8b5d-5818e5dedd04","Type":"ContainerStarted","Data":"55b1d11631e84593e2fd939ec9ffc86886bddb6702e82e0a9bc3970a84d53d83"}
Sep 15 21:00:03.607349 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:03.607273    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9" podStartSLOduration=1.071405537 podStartE2EDuration="3.60725822s" totalImagesPullingTime="2.535852683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:03.606205304 +0000 UTC m=+1394.393329323" watchObservedRunningTime="2026-09-15 21:00:03.60725822 +0000 UTC m=+1394.394382289"
Sep 15 21:00:04.599506 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:04.599468    2579 generic.go:356] "Generic (PLEG): container finished" podID="05c15548-5553-4c41-8b5d-5818e5dedd04" containerID="55b1d11631e84593e2fd939ec9ffc86886bddb6702e82e0a9bc3970a84d53d83" exitCode=0
Sep 15 21:00:04.599918 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:04.599558    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9" event={"ID":"05c15548-5553-4c41-8b5d-5818e5dedd04","Type":"ContainerDied","Data":"55b1d11631e84593e2fd939ec9ffc86886bddb6702e82e0a9bc3970a84d53d83"}
Sep 15 21:00:05.752149 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:05.752125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"
Sep 15 21:00:06.273990 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.273954    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:00:06.274570 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.274554    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c15548-5553-4c41-8b5d-5818e5dedd04" containerName="cleanup"
Sep 15 21:00:06.274616 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.274575    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c15548-5553-4c41-8b5d-5818e5dedd04" containerName="cleanup"
Sep 15 21:00:06.274683 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.274673    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c15548-5553-4c41-8b5d-5818e5dedd04" containerName="cleanup"
Sep 15 21:00:06.278321 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.278303    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.281108 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.281088    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-3825fc\""
Sep 15 21:00:06.287931 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.287905    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:00:06.404881 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.404853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.404881 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.404885    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbmq\" (UniqueName: \"kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.508634 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.508599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.508911 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.508890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbmq\" (UniqueName: \"kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.513524 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.510378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.518043 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.517988    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbmq\" (UniqueName: \"kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq\") pod \"payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.590290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.590206    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:06.611992 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.611962    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"
Sep 15 21:00:06.611992 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.611984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9" event={"ID":"05c15548-5553-4c41-8b5d-5818e5dedd04","Type":"ContainerDied","Data":"2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1"}
Sep 15 21:00:06.612187 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.612059    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a9f341f2e1a9622b9e1103285263ad40746ffe769df4f5be68e3c3b405ae2d1"
Sep 15 21:00:06.866211 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:06.866182    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:00:06.868447 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:00:06.868397    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe32c0f_f416_4a69_83bd_4d1958bf63ea.slice/crio-0795c684f5fe72e917d104085f7d3064cda3095a6fbb4bfa2c8d7d4ee79c72d9 WatchSource:0}: Error finding container 0795c684f5fe72e917d104085f7d3064cda3095a6fbb4bfa2c8d7d4ee79c72d9: Status 404 returned error can't find the container with id 0795c684f5fe72e917d104085f7d3064cda3095a6fbb4bfa2c8d7d4ee79c72d9
Sep 15 21:00:07.619216 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:07.619179    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" event={"ID":"cfe32c0f-f416-4a69-83bd-4d1958bf63ea","Type":"ContainerStarted","Data":"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"}
Sep 15 21:00:07.619526 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:07.619342    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:07.619526 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:07.619377    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" event={"ID":"cfe32c0f-f416-4a69-83bd-4d1958bf63ea","Type":"ContainerStarted","Data":"0795c684f5fe72e917d104085f7d3064cda3095a6fbb4bfa2c8d7d4ee79c72d9"}
Sep 15 21:00:07.637831 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:07.637783    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" podStartSLOduration=1.637766957 podStartE2EDuration="1.637766957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:07.636767004 +0000 UTC m=+1398.423891035" watchObservedRunningTime="2026-09-15 21:00:07.637766957 +0000 UTC m=+1398.424890985"
Sep 15 21:00:18.472178 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:18.472140    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:00:18.626964 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:18.626935    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:00:30.059647 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:30.059615    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 21:00:45.079345 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.079249    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" podUID="4e94a744-b422-4c3d-bf4f-be619367f375" containerName="main" containerID="cri-o://06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6" gracePeriod=45
Sep 15 21:00:45.332023 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.331944    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 21:00:45.470883 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470850    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.470883 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470880    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.471235 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.471235 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470936    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.471235 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470956    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5h6jr\" (UniqueName: \"kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.471235 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.470987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs\") pod \"4e94a744-b422-4c3d-bf4f-be619367f375\" (UID: \"4e94a744-b422-4c3d-bf4f-be619367f375\") "
Sep 15 21:00:45.471432 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.471238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home" (OuterVolumeSpecName: "home") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:00:45.471432 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.471301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:00:45.471432 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.471344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:00:45.473414 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.473394    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm" (OuterVolumeSpecName: "dshm") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:00:45.473498 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.473438    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr" (OuterVolumeSpecName: "kube-api-access-5h6jr") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "kube-api-access-5h6jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:00:45.473498 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.473484    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e94a744-b422-4c3d-bf4f-be619367f375" (UID: "4e94a744-b422-4c3d-bf4f-be619367f375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:00:45.572335 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572303    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.572335 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572335    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.572536 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572349    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.572536 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572358    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e94a744-b422-4c3d-bf4f-be619367f375-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.572536 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572366    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h6jr\" (UniqueName: \"kubernetes.io/projected/4e94a744-b422-4c3d-bf4f-be619367f375-kube-api-access-5h6jr\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.572536 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.572376    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e94a744-b422-4c3d-bf4f-be619367f375-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:00:45.840611 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.840577    2579 generic.go:356] "Generic (PLEG): container finished" podID="4e94a744-b422-4c3d-bf4f-be619367f375" containerID="06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6" exitCode=0
Sep 15 21:00:45.840774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.840619    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"
Sep 15 21:00:45.840774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.840658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" event={"ID":"4e94a744-b422-4c3d-bf4f-be619367f375","Type":"ContainerDied","Data":"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"}
Sep 15 21:00:45.840774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.840702    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r" event={"ID":"4e94a744-b422-4c3d-bf4f-be619367f375","Type":"ContainerDied","Data":"3ac10d8f3efbea1b53e338171319937627c8ff698a4e3ed990cfda4a28670e06"}
Sep 15 21:00:45.840774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.840719    2579 scope.go:117] "RemoveContainer" containerID="06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"
Sep 15 21:00:45.849479 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.849455    2579 scope.go:117] "RemoveContainer" containerID="06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"
Sep 15 21:00:45.849771 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:00:45.849751    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6\": container with ID starting with 06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6 not found: ID does not exist" containerID="06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"
Sep 15 21:00:45.849820 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.849781    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6"} err="failed to get container status \"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6\": rpc error: code = NotFound desc = could not find container \"06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6\": container with ID starting with 06cb05c089ac81ace552c3b7be8439187c8deaf409e7b34437f4d7235a7218d6 not found: ID does not exist"
Sep 15 21:00:45.857370 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.857347    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 21:00:45.861555 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:45.861536    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d5b5b9/test-model-d5b5b9-kserve-b8ccfbd7-7mp9r"]
Sep 15 21:00:47.669325 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:47.669295    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e94a744-b422-4c3d-bf4f-be619367f375" path="/var/lib/kubelet/pods/4e94a744-b422-4c3d-bf4f-be619367f375/volumes"
Sep 15 21:00:49.021938 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.021903    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:00:49.022625 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.022603    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e94a744-b422-4c3d-bf4f-be619367f375" containerName="main"
Sep 15 21:00:49.022680 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.022630    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e94a744-b422-4c3d-bf4f-be619367f375" containerName="main"
Sep 15 21:00:49.022774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.022761    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e94a744-b422-4c3d-bf4f-be619367f375" containerName="main"
Sep 15 21:00:49.026290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.026270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.029201 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.029180    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-65001159-serving-cert\""
Sep 15 21:00:49.035810 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.035785    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:00:49.105214 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.105177    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.105392 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.105238    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.105392 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.105294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.105465 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.105391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc65b\" (UniqueName: \"kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.206313 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.206277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.206488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.206338    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc65b\" (UniqueName: \"kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.206488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.206375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.206488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.206431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.206651 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:00:49.206543    2579 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-65001159-serving-cert: secret "maas-api-e2e-mt-65001159-serving-cert" not found
Sep 15 21:00:49.206651 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:00:49.206622    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls podName:c8c92913-b961-4473-ae88-695c74d56249 nodeName:}" failed. No retries permitted until 2026-09-15 21:00:49.70660287 +0000 UTC m=+1440.493726877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls") pod "maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" (UID: "c8c92913-b961-4473-ae88-695c74d56249") : secret "maas-api-e2e-mt-65001159-serving-cert" not found
Sep 15 21:00:49.207146 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.207122    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.208884 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.208861    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.215487 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.215469    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc65b\" (UniqueName: \"kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.710375 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.710339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.712774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.712754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") pod \"maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:49.786035 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.785978    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:00:49.790643 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.790624    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:49.793761 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.793741    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-65001159\""
Sep 15 21:00:49.795237 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.795221    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-65001159-dockercfg-kj4n8\""
Sep 15 21:00:49.805380 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.805356    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:00:49.912649 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.912614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:49.912800 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.912723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5hs5\" (UniqueName: \"kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:49.941109 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:49.941069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:50.013434 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.013322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5hs5\" (UniqueName: \"kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.013686 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.013488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.014347 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.014323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.023474 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.023451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5hs5\" (UniqueName: \"kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5\") pod \"payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.100983 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.100944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.230650 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.230626    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:00:50.232000 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:00:50.231974    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c92913_b961_4473_ae88_695c74d56249.slice/crio-e40845d043e60b2317c43c320e34ab172bf6bb6639b1edb3e0a8941e008b3af9 WatchSource:0}: Error finding container e40845d043e60b2317c43c320e34ab172bf6bb6639b1edb3e0a8941e008b3af9: Status 404 returned error can't find the container with id e40845d043e60b2317c43c320e34ab172bf6bb6639b1edb3e0a8941e008b3af9
Sep 15 21:00:50.398339 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.398315    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:00:50.401182 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:00:50.401147    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2275864e_47a7_4bf5_bfec_51d2624d5030.slice/crio-4ff24052ecbfe0d3bc0e4b0feebd27e72ea63eacd65e035a52597698357ca821 WatchSource:0}: Error finding container 4ff24052ecbfe0d3bc0e4b0feebd27e72ea63eacd65e035a52597698357ca821: Status 404 returned error can't find the container with id 4ff24052ecbfe0d3bc0e4b0feebd27e72ea63eacd65e035a52597698357ca821
Sep 15 21:00:50.876979 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.876943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" event={"ID":"c8c92913-b961-4473-ae88-695c74d56249","Type":"ContainerStarted","Data":"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"}
Sep 15 21:00:50.877190 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.877023    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:00:50.877190 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.877048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" event={"ID":"c8c92913-b961-4473-ae88-695c74d56249","Type":"ContainerStarted","Data":"e40845d043e60b2317c43c320e34ab172bf6bb6639b1edb3e0a8941e008b3af9"}
Sep 15 21:00:50.878721 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.878699    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" event={"ID":"2275864e-47a7-4bf5-bfec-51d2624d5030","Type":"ContainerStarted","Data":"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"}
Sep 15 21:00:50.878840 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.878746    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" event={"ID":"2275864e-47a7-4bf5-bfec-51d2624d5030","Type":"ContainerStarted","Data":"4ff24052ecbfe0d3bc0e4b0feebd27e72ea63eacd65e035a52597698357ca821"}
Sep 15 21:00:50.878840 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.878775    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:00:50.899497 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.899448    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" podStartSLOduration=2.393863953 podStartE2EDuration="2.899431877s" totalImagesPullingTime="505.567924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:50.898494687 +0000 UTC m=+1441.685618718" watchObservedRunningTime="2026-09-15 21:00:50.899431877 +0000 UTC m=+1441.686555930"
Sep 15 21:00:50.918704 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:50.918656    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" podStartSLOduration=1.918640845 podStartE2EDuration="1.918640845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:00:50.917986859 +0000 UTC m=+1441.705110888" watchObservedRunningTime="2026-09-15 21:00:50.918640845 +0000 UTC m=+1441.705764874"
Sep 15 21:00:51.773587 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:51.773547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:00:56.889449 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:00:56.889418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:01:01.885582 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:01.885552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:01:12.751607 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.751557    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 21:01:12.752097 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.751811    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" podUID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" containerName="maas-api" containerID="cri-o://a468c3c9257374c875337462c5cdeb34ee9116557f4cd2943df79d099a088244" gracePeriod=30
Sep 15 21:01:12.780962 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.780933    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"]
Sep 15 21:01:12.782907 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.782884    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d5b5b9-29825100-kxvj9"]
Sep 15 21:01:12.811512 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.811477    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 21:01:12.811817 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:12.811786    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" podUID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" containerName="payload-pre-processing" containerID="cri-o://31d2e76331b195fc053a98dfb0961065d8de610eefa36876e119e4d6ce8908fa" gracePeriod=30
Sep 15 21:01:13.004116 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.004020    2579 generic.go:356] "Generic (PLEG): container finished" podID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" containerID="31d2e76331b195fc053a98dfb0961065d8de610eefa36876e119e4d6ce8908fa" exitCode=0
Sep 15 21:01:13.004116 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.004093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" event={"ID":"55bd40cc-61f9-4001-acea-1b0cd11dc96d","Type":"ContainerDied","Data":"31d2e76331b195fc053a98dfb0961065d8de610eefa36876e119e4d6ce8908fa"}
Sep 15 21:01:13.006429 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.006389    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" containerID="a468c3c9257374c875337462c5cdeb34ee9116557f4cd2943df79d099a088244" exitCode=0
Sep 15 21:01:13.006586 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.006550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" event={"ID":"1a0be28f-5dd6-4813-949d-ca07fa5680a8","Type":"ContainerDied","Data":"a468c3c9257374c875337462c5cdeb34ee9116557f4cd2943df79d099a088244"}
Sep 15 21:01:13.081764 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.081739    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 21:01:13.113628 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.113593    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 21:01:13.139836 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.139802    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume\") pod \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") "
Sep 15 21:01:13.140057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.139856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwp6h\" (UniqueName: \"kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h\") pod \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\" (UID: \"55bd40cc-61f9-4001-acea-1b0cd11dc96d\") "
Sep 15 21:01:13.140057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.139907    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls\") pod \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") "
Sep 15 21:01:13.140057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.139937    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs\") pod \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") "
Sep 15 21:01:13.140057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.139993    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh\") pod \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") "
Sep 15 21:01:13.140310 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.140110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca\") pod \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\" (UID: \"1a0be28f-5dd6-4813-949d-ca07fa5680a8\") "
Sep 15 21:01:13.140676 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.140647    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55bd40cc-61f9-4001-acea-1b0cd11dc96d" (UID: "55bd40cc-61f9-4001-acea-1b0cd11dc96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:01:13.140926 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.140890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1a0be28f-5dd6-4813-949d-ca07fa5680a8" (UID: "1a0be28f-5dd6-4813-949d-ca07fa5680a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:01:13.142621 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.142593    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h" (OuterVolumeSpecName: "kube-api-access-cwp6h") pod "55bd40cc-61f9-4001-acea-1b0cd11dc96d" (UID: "55bd40cc-61f9-4001-acea-1b0cd11dc96d"). InnerVolumeSpecName "kube-api-access-cwp6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:01:13.142948 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.142922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh" (OuterVolumeSpecName: "kube-api-access-llgbh") pod "1a0be28f-5dd6-4813-949d-ca07fa5680a8" (UID: "1a0be28f-5dd6-4813-949d-ca07fa5680a8"). InnerVolumeSpecName "kube-api-access-llgbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:01:13.143122 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.143101    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1a0be28f-5dd6-4813-949d-ca07fa5680a8" (UID: "1a0be28f-5dd6-4813-949d-ca07fa5680a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:01:13.144678 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.144657    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1a0be28f-5dd6-4813-949d-ca07fa5680a8" (UID: "1a0be28f-5dd6-4813-949d-ca07fa5680a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:01:13.241590 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241550    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-maas-api-tls\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.241590 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241577    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a0be28f-5dd6-4813-949d-ca07fa5680a8-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.241590 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241586    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llgbh\" (UniqueName: \"kubernetes.io/projected/1a0be28f-5dd6-4813-949d-ca07fa5680a8-kube-api-access-llgbh\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.241590 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241596    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a0be28f-5dd6-4813-949d-ca07fa5680a8-openshift-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.241590 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241606    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55bd40cc-61f9-4001-acea-1b0cd11dc96d-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.241996 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.241615    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwp6h\" (UniqueName: \"kubernetes.io/projected/55bd40cc-61f9-4001-acea-1b0cd11dc96d-kube-api-access-cwp6h\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:13.669407 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:13.669376    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c15548-5553-4c41-8b5d-5818e5dedd04" path="/var/lib/kubelet/pods/05c15548-5553-4c41-8b5d-5818e5dedd04/volumes"
Sep 15 21:01:14.013760 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.013680    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"
Sep 15 21:01:14.014260 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.013680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9" event={"ID":"55bd40cc-61f9-4001-acea-1b0cd11dc96d","Type":"ContainerDied","Data":"f15f9d8ada652e9de0299192967b441a5d8138f5a99c9e72671b30961b09dbb7"}
Sep 15 21:01:14.014260 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.013848    2579 scope.go:117] "RemoveContainer" containerID="31d2e76331b195fc053a98dfb0961065d8de610eefa36876e119e4d6ce8908fa"
Sep 15 21:01:14.015457 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.015316    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng" event={"ID":"1a0be28f-5dd6-4813-949d-ca07fa5680a8","Type":"ContainerDied","Data":"ac0a4b0105fb4c96cd1a5aef85df2bcf66f4006ace3f54878ea67ecbe5f65069"}
Sep 15 21:01:14.015457 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.015348    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"
Sep 15 21:01:14.023419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.023399    2579 scope.go:117] "RemoveContainer" containerID="a468c3c9257374c875337462c5cdeb34ee9116557f4cd2943df79d099a088244"
Sep 15 21:01:14.037520 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.037496    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 21:01:14.042947 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.042926    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d5b5b9-65bc966586-jp9z9"]
Sep 15 21:01:14.057674 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.057653    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 21:01:14.062320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:14.062301    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-d5b5b9-74fdfd99cd-s89ng"]
Sep 15 21:01:15.669804 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:15.669768    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" path="/var/lib/kubelet/pods/1a0be28f-5dd6-4813-949d-ca07fa5680a8/volumes"
Sep 15 21:01:15.670189 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:15.670133    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" path="/var/lib/kubelet/pods/55bd40cc-61f9-4001-acea-1b0cd11dc96d/volumes"
Sep 15 21:01:22.275272 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.275230    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:01:22.765167 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:01:22.765783 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765763    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" containerName="maas-api"
Sep 15 21:01:22.765841 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765789    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" containerName="maas-api"
Sep 15 21:01:22.765841 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765811    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" containerName="payload-pre-processing"
Sep 15 21:01:22.765841 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765820    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" containerName="payload-pre-processing"
Sep 15 21:01:22.765949 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765923    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="55bd40cc-61f9-4001-acea-1b0cd11dc96d" containerName="payload-pre-processing"
Sep 15 21:01:22.765949 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.765937    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a0be28f-5dd6-4813-949d-ca07fa5680a8" containerName="maas-api"
Sep 15 21:01:22.770119 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.770096    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.773560 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.773526    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"kube-root-ca.crt\""
Sep 15 21:01:22.774928 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.774903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"e2e-conflict-b-466b1a2b-kserve-self-signed-certs\""
Sep 15 21:01:22.775349 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.775329    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"default-dockercfg-sljmb\""
Sep 15 21:01:22.775697 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.775677    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-16bde4\"/\"openshift-service-ca.crt\""
Sep 15 21:01:22.778878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.778852    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:01:22.835814 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.835775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v94b\" (UniqueName: \"kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.835991 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.835825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.835991 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.835882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.835991 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.835971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.836135 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.836000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.836135 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.836080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937426 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v94b\" (UniqueName: \"kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937426 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937497    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937924 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937901    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.937990 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937910    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.938101 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.937999    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.939875 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.939852    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.940133 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.940116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:22.946682 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:22.946656    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v94b\" (UniqueName: \"kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b\") pod \"e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") " pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:23.084043 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:23.083954    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:23.357289 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:23.357264    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:01:23.359385 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:01:23.359360    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0cbbdc9_8d68_4827_b3b9_512648b86028.slice/crio-22d02a2b197e230a98fab94664b2937d75e9ca3e1dc6bffd881e2930afa1beac WatchSource:0}: Error finding container 22d02a2b197e230a98fab94664b2937d75e9ca3e1dc6bffd881e2930afa1beac: Status 404 returned error can't find the container with id 22d02a2b197e230a98fab94664b2937d75e9ca3e1dc6bffd881e2930afa1beac
Sep 15 21:01:24.077758 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:24.077720    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" event={"ID":"a0cbbdc9-8d68-4827-b3b9-512648b86028","Type":"ContainerStarted","Data":"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"}
Sep 15 21:01:24.077758 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:24.077763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" event={"ID":"a0cbbdc9-8d68-4827-b3b9-512648b86028","Type":"ContainerStarted","Data":"22d02a2b197e230a98fab94664b2937d75e9ca3e1dc6bffd881e2930afa1beac"}
Sep 15 21:01:24.094801 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:24.094749    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" podStartSLOduration=2.094734077 podStartE2EDuration="2.094734077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:01:24.092539062 +0000 UTC m=+1474.879663093" watchObservedRunningTime="2026-09-15 21:01:24.094734077 +0000 UTC m=+1474.881858105"
Sep 15 21:01:30.326292 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.326260    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 21:01:30.326767 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.326508    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" podUID="9fd92500-8430-4f72-8585-0d3d82cdf42c" containerName="payload-pre-processing" containerID="cri-o://6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682" gracePeriod=30
Sep 15 21:01:30.585146 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.585078    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 21:01:30.709139 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.709101    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume\") pod \"9fd92500-8430-4f72-8585-0d3d82cdf42c\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") "
Sep 15 21:01:30.709339 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.709185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct6zp\" (UniqueName: \"kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp\") pod \"9fd92500-8430-4f72-8585-0d3d82cdf42c\" (UID: \"9fd92500-8430-4f72-8585-0d3d82cdf42c\") "
Sep 15 21:01:30.709491 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.709464    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9fd92500-8430-4f72-8585-0d3d82cdf42c" (UID: "9fd92500-8430-4f72-8585-0d3d82cdf42c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:01:30.711582 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.711553    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp" (OuterVolumeSpecName: "kube-api-access-ct6zp") pod "9fd92500-8430-4f72-8585-0d3d82cdf42c" (UID: "9fd92500-8430-4f72-8585-0d3d82cdf42c"). InnerVolumeSpecName "kube-api-access-ct6zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:01:30.810702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.810663    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fd92500-8430-4f72-8585-0d3d82cdf42c-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:30.810702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:30.810694    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct6zp\" (UniqueName: \"kubernetes.io/projected/9fd92500-8430-4f72-8585-0d3d82cdf42c-kube-api-access-ct6zp\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:01:31.124453 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.124412    2579 generic.go:356] "Generic (PLEG): container finished" podID="9fd92500-8430-4f72-8585-0d3d82cdf42c" containerID="6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682" exitCode=0
Sep 15 21:01:31.124630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.124475    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"
Sep 15 21:01:31.124630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.124493    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" event={"ID":"9fd92500-8430-4f72-8585-0d3d82cdf42c","Type":"ContainerDied","Data":"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"}
Sep 15 21:01:31.124630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.124578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb" event={"ID":"9fd92500-8430-4f72-8585-0d3d82cdf42c","Type":"ContainerDied","Data":"c76c50bd3ea892f22bb1d1e192933919e9db44447fbfe08de946046db72a3d03"}
Sep 15 21:01:31.124630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.124603    2579 scope.go:117] "RemoveContainer" containerID="6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"
Sep 15 21:01:31.134186 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.134161    2579 scope.go:117] "RemoveContainer" containerID="6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"
Sep 15 21:01:31.134483 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:01:31.134463    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682\": container with ID starting with 6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682 not found: ID does not exist" containerID="6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"
Sep 15 21:01:31.134541 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.134495    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682"} err="failed to get container status \"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682\": rpc error: code = NotFound desc = could not find container \"6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682\": container with ID starting with 6d738f32044723aec030935352d2a8c0c9c27ffb9f7234bf0594675992dbd682 not found: ID does not exist"
Sep 15 21:01:31.146404 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.146376    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 21:01:31.150734 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.150708    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-988a09-5c8f77654b-9pmrb"]
Sep 15 21:01:31.668344 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:31.668308    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fd92500-8430-4f72-8585-0d3d82cdf42c" path="/var/lib/kubelet/pods/9fd92500-8430-4f72-8585-0d3d82cdf42c/volumes"
Sep 15 21:01:33.085053 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:33.085024    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:33.085563 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:33.085138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:33.098200 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:33.098175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:33.147598 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:33.147572    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:01:49.751146 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:49.751116    2579 scope.go:117] "RemoveContainer" containerID="77138d50c14f269b2a2f0f1e9ba4bb9ef0ab2ede34aedb7447db4b7578bf4cbe"
Sep 15 21:01:53.073484 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:01:53.073448    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:02:25.358499 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.358465    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:02:25.358998 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.358860    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fd92500-8430-4f72-8585-0d3d82cdf42c" containerName="payload-pre-processing"
Sep 15 21:02:25.358998 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.358871    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fd92500-8430-4f72-8585-0d3d82cdf42c" containerName="payload-pre-processing"
Sep 15 21:02:25.358998 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.358946    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fd92500-8430-4f72-8585-0d3d82cdf42c" containerName="payload-pre-processing"
Sep 15 21:02:25.362221 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.362203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.364972 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.364953    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-728877\""
Sep 15 21:02:25.373550 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.373523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:02:25.392940 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.392901    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:02:25.396278 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.396257    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.398983 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.398957    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-728877-dockercfg-f8x5j\""
Sep 15 21:02:25.405481 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.405460    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:02:25.523296 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.523254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.523296 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.523292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dbgp\" (UniqueName: \"kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.523570 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.523340    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.523570 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.523457    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4klb\" (UniqueName: \"kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.624962 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.624924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.625187 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.624971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dbgp\" (UniqueName: \"kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.625187 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.625071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.625187 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.625170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4klb\" (UniqueName: \"kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.625634 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.625610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.625733 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.625716    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.634387 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.634360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dbgp\" (UniqueName: \"kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp\") pod \"payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.634387 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.634362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4klb\" (UniqueName: \"kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb\") pod \"payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.673867 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.673833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:25.710180 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.710148    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:25.951171 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.951150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:02:25.953202 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:02:25.953176    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a2e6c1_73c3_4553_8692_139b29664c0f.slice/crio-f6592c8b3ab34f3427fe993f70b43077e345b78195eb0ef7bf3f46704bdbf5f8 WatchSource:0}: Error finding container f6592c8b3ab34f3427fe993f70b43077e345b78195eb0ef7bf3f46704bdbf5f8: Status 404 returned error can't find the container with id f6592c8b3ab34f3427fe993f70b43077e345b78195eb0ef7bf3f46704bdbf5f8
Sep 15 21:02:25.992495 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:25.992467    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:02:25.994298 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:02:25.994268    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43cf0e1_7b6e_4f1a_ac89_e40a6f5d08df.slice/crio-937c7017b1b89c021c1d1311f7ffe618a7fd3349142c5e4adb7083a263cface3 WatchSource:0}: Error finding container 937c7017b1b89c021c1d1311f7ffe618a7fd3349142c5e4adb7083a263cface3: Status 404 returned error can't find the container with id 937c7017b1b89c021c1d1311f7ffe618a7fd3349142c5e4adb7083a263cface3
Sep 15 21:02:26.425870 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.425836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" event={"ID":"39a2e6c1-73c3-4553-8692-139b29664c0f","Type":"ContainerStarted","Data":"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"}
Sep 15 21:02:26.426328 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.425889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" event={"ID":"39a2e6c1-73c3-4553-8692-139b29664c0f","Type":"ContainerStarted","Data":"f6592c8b3ab34f3427fe993f70b43077e345b78195eb0ef7bf3f46704bdbf5f8"}
Sep 15 21:02:26.426328 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.425949    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:26.427524 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.427501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" event={"ID":"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df","Type":"ContainerStarted","Data":"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"}
Sep 15 21:02:26.427672 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.427542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" event={"ID":"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df","Type":"ContainerStarted","Data":"937c7017b1b89c021c1d1311f7ffe618a7fd3349142c5e4adb7083a263cface3"}
Sep 15 21:02:26.427672 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.427569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:02:26.444771 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.444715    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" podStartSLOduration=1.444697308 podStartE2EDuration="1.444697308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:26.441117697 +0000 UTC m=+1537.228241726" watchObservedRunningTime="2026-09-15 21:02:26.444697308 +0000 UTC m=+1537.231821338"
Sep 15 21:02:26.458781 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:26.458725    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" podStartSLOduration=1.458704375 podStartE2EDuration="1.458704375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:02:26.455839702 +0000 UTC m=+1537.242963731" watchObservedRunningTime="2026-09-15 21:02:26.458704375 +0000 UTC m=+1537.245828406"
Sep 15 21:02:35.323284 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:35.323248    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:02:37.434582 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:37.434553    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:02:37.434986 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:02:37.434620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:03:16.369063 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.369026    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:03:16.373936 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.373918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.376578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.376561    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3f1c93\"/\"distinct-2-w0-3f1c93-kserve-self-signed-certs\""
Sep 15 21:03:16.383555 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.383530    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:03:16.419419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.419578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.419578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419496    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.419578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.419578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419547    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.419719 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.419613    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv27q\" (UniqueName: \"kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520445 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520645 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520645 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520645 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520826 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520826 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv27q\" (UniqueName: \"kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520826 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.520988 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.520955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.521084 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.521054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.522904 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.522885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.523318 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.523296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.528661 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.528641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv27q\" (UniqueName: \"kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q\") pod \"distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") " pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.683944 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.683904    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:16.956589 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:16.956563    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:03:16.958378 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:03:16.958340    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9920da0_0fd3_4003_8604_ebd479f34c2d.slice/crio-68a3a3eecbcde83852faa2b8a325dc28d2a892b106911d75153898ea82fadc8c WatchSource:0}: Error finding container 68a3a3eecbcde83852faa2b8a325dc28d2a892b106911d75153898ea82fadc8c: Status 404 returned error can't find the container with id 68a3a3eecbcde83852faa2b8a325dc28d2a892b106911d75153898ea82fadc8c
Sep 15 21:03:17.696256 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:17.696223    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" event={"ID":"f9920da0-0fd3-4003-8604-ebd479f34c2d","Type":"ContainerStarted","Data":"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"}
Sep 15 21:03:17.696256 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:17.696262    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" event={"ID":"f9920da0-0fd3-4003-8604-ebd479f34c2d","Type":"ContainerStarted","Data":"68a3a3eecbcde83852faa2b8a325dc28d2a892b106911d75153898ea82fadc8c"}
Sep 15 21:03:17.714280 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:17.714238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" podStartSLOduration=1.714223378 podStartE2EDuration="1.714223378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:03:17.713290496 +0000 UTC m=+1588.500414525" watchObservedRunningTime="2026-09-15 21:03:17.714223378 +0000 UTC m=+1588.501347444"
Sep 15 21:03:26.684048 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:26.683977    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:26.684451 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:26.684065    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:26.696871 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:26.696848    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:26.760834 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:26.760800    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:03:26.770326 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:26.770301    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:03:36.872307 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:36.872277    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:03:51.892820 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:51.892772    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" podUID="a0cbbdc9-8d68-4827-b3b9-512648b86028" containerName="main" containerID="cri-o://6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10" gracePeriod=45
Sep 15 21:03:52.148799 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.148747    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:03:52.158553 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158524    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4v94b\" (UniqueName: \"kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158553 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158563    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158588    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158612    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158689    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache\") pod \"a0cbbdc9-8d68-4827-b3b9-512648b86028\" (UID: \"a0cbbdc9-8d68-4827-b3b9-512648b86028\") "
Sep 15 21:03:52.158943 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home" (OuterVolumeSpecName: "home") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:03:52.158943 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158857    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:03:52.159054 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158972    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.159054 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.158990    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.159157 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.159054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:03:52.160880 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.160854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm" (OuterVolumeSpecName: "dshm") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:03:52.160987 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.160938    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b" (OuterVolumeSpecName: "kube-api-access-4v94b") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "kube-api-access-4v94b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:03:52.161080 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.161030    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0cbbdc9-8d68-4827-b3b9-512648b86028" (UID: "a0cbbdc9-8d68-4827-b3b9-512648b86028"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:03:52.259487 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.259447    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.259487 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.259485    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0cbbdc9-8d68-4827-b3b9-512648b86028-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.259487 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.259497    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v94b\" (UniqueName: \"kubernetes.io/projected/a0cbbdc9-8d68-4827-b3b9-512648b86028-kube-api-access-4v94b\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.259713 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.259507    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0cbbdc9-8d68-4827-b3b9-512648b86028-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:03:52.889752 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.889718    2579 generic.go:356] "Generic (PLEG): container finished" podID="a0cbbdc9-8d68-4827-b3b9-512648b86028" containerID="6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10" exitCode=0
Sep 15 21:03:52.889945 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.889779    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"
Sep 15 21:03:52.889945 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.889812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" event={"ID":"a0cbbdc9-8d68-4827-b3b9-512648b86028","Type":"ContainerDied","Data":"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"}
Sep 15 21:03:52.889945 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.889857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx" event={"ID":"a0cbbdc9-8d68-4827-b3b9-512648b86028","Type":"ContainerDied","Data":"22d02a2b197e230a98fab94664b2937d75e9ca3e1dc6bffd881e2930afa1beac"}
Sep 15 21:03:52.889945 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.889880    2579 scope.go:117] "RemoveContainer" containerID="6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"
Sep 15 21:03:52.900357 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.900196    2579 scope.go:117] "RemoveContainer" containerID="6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"
Sep 15 21:03:52.900656 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:03:52.900511    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10\": container with ID starting with 6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10 not found: ID does not exist" containerID="6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"
Sep 15 21:03:52.900656 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.900550    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10"} err="failed to get container status \"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10\": rpc error: code = NotFound desc = could not find container \"6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10\": container with ID starting with 6552960dc54a59c6614047a7d80ce9edf32e8d4bb2b2369efa3f678cc09ddc10 not found: ID does not exist"
Sep 15 21:03:52.912531 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.912508    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:03:52.916137 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:52.916118    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-16bde4/e2e-conflict-b-466b1a2b-kserve-749599559f-k6xnx"]
Sep 15 21:03:53.669568 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:03:53.669535    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0cbbdc9-8d68-4827-b3b9-512648b86028" path="/var/lib/kubelet/pods/a0cbbdc9-8d68-4827-b3b9-512648b86028/volumes"
Sep 15 21:04:03.971793 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:03.971760    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:04:08.245947 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.245918    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:04:08.246427 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.246188    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" podUID="c8c92913-b961-4473-ae88-695c74d56249" containerName="maas-api" containerID="cri-o://6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed" gracePeriod=30
Sep 15 21:04:08.294912 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.294872    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:04:08.295435 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.295192    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" podUID="2275864e-47a7-4bf5-bfec-51d2624d5030" containerName="payload-pre-processing" containerID="cri-o://626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed" gracePeriod=30
Sep 15 21:04:08.558964 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.558934    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:04:08.595551 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.595522    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:04:08.622140 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622114    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") pod \"c8c92913-b961-4473-ae88-695c74d56249\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") "
Sep 15 21:04:08.622291 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622158    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc65b\" (UniqueName: \"kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b\") pod \"c8c92913-b961-4473-ae88-695c74d56249\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") "
Sep 15 21:04:08.622291 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622194    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca\") pod \"c8c92913-b961-4473-ae88-695c74d56249\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") "
Sep 15 21:04:08.622414 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622292    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs\") pod \"c8c92913-b961-4473-ae88-695c74d56249\" (UID: \"c8c92913-b961-4473-ae88-695c74d56249\") "
Sep 15 21:04:08.622594 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622555    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8c92913-b961-4473-ae88-695c74d56249" (UID: "c8c92913-b961-4473-ae88-695c74d56249"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:04:08.622738 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.622695    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c92913-b961-4473-ae88-695c74d56249-openshift-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.624437 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.624411    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8c92913-b961-4473-ae88-695c74d56249" (UID: "c8c92913-b961-4473-ae88-695c74d56249"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:04:08.624552 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.624468    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8c92913-b961-4473-ae88-695c74d56249" (UID: "c8c92913-b961-4473-ae88-695c74d56249"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:04:08.624807 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.624790    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b" (OuterVolumeSpecName: "kube-api-access-lc65b") pod "c8c92913-b961-4473-ae88-695c74d56249" (UID: "c8c92913-b961-4473-ae88-695c74d56249"). InnerVolumeSpecName "kube-api-access-lc65b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:08.723962 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.723933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume\") pod \"2275864e-47a7-4bf5-bfec-51d2624d5030\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") "
Sep 15 21:04:08.724152 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.724031    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5hs5\" (UniqueName: \"kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5\") pod \"2275864e-47a7-4bf5-bfec-51d2624d5030\" (UID: \"2275864e-47a7-4bf5-bfec-51d2624d5030\") "
Sep 15 21:04:08.724207 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.724196    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-maas-api-tls\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.724250 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.724210    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc65b\" (UniqueName: \"kubernetes.io/projected/c8c92913-b961-4473-ae88-695c74d56249-kube-api-access-lc65b\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.724250 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.724219    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8c92913-b961-4473-ae88-695c74d56249-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.724417 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.724392    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2275864e-47a7-4bf5-bfec-51d2624d5030" (UID: "2275864e-47a7-4bf5-bfec-51d2624d5030"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:04:08.726217 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.726198    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5" (OuterVolumeSpecName: "kube-api-access-b5hs5") pod "2275864e-47a7-4bf5-bfec-51d2624d5030" (UID: "2275864e-47a7-4bf5-bfec-51d2624d5030"). InnerVolumeSpecName "kube-api-access-b5hs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:04:08.825364 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.825279    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2275864e-47a7-4bf5-bfec-51d2624d5030-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.825364 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.825306    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5hs5\" (UniqueName: \"kubernetes.io/projected/2275864e-47a7-4bf5-bfec-51d2624d5030-kube-api-access-b5hs5\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:04:08.980058 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.980021    2579 generic.go:356] "Generic (PLEG): container finished" podID="2275864e-47a7-4bf5-bfec-51d2624d5030" containerID="626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed" exitCode=0
Sep 15 21:04:08.980254 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.980080    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"
Sep 15 21:04:08.980254 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.980114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" event={"ID":"2275864e-47a7-4bf5-bfec-51d2624d5030","Type":"ContainerDied","Data":"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"}
Sep 15 21:04:08.980254 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.980172    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk" event={"ID":"2275864e-47a7-4bf5-bfec-51d2624d5030","Type":"ContainerDied","Data":"4ff24052ecbfe0d3bc0e4b0feebd27e72ea63eacd65e035a52597698357ca821"}
Sep 15 21:04:08.980254 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.980194    2579 scope.go:117] "RemoveContainer" containerID="626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"
Sep 15 21:04:08.984940 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.984916    2579 generic.go:356] "Generic (PLEG): container finished" podID="c8c92913-b961-4473-ae88-695c74d56249" containerID="6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed" exitCode=0
Sep 15 21:04:08.985051 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.984967    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"
Sep 15 21:04:08.985051 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.985027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" event={"ID":"c8c92913-b961-4473-ae88-695c74d56249","Type":"ContainerDied","Data":"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"}
Sep 15 21:04:08.985163 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.985070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2" event={"ID":"c8c92913-b961-4473-ae88-695c74d56249","Type":"ContainerDied","Data":"e40845d043e60b2317c43c320e34ab172bf6bb6639b1edb3e0a8941e008b3af9"}
Sep 15 21:04:08.994064 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.994048    2579 scope.go:117] "RemoveContainer" containerID="626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"
Sep 15 21:04:08.994339 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:04:08.994321    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed\": container with ID starting with 626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed not found: ID does not exist" containerID="626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"
Sep 15 21:04:08.994413 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.994346    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed"} err="failed to get container status \"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed\": rpc error: code = NotFound desc = could not find container \"626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed\": container with ID starting with 626c7e81f837c260bf1cdffe892626a32f5ae54ab436b67c6f1227f3613fcbed not found: ID does not exist"
Sep 15 21:04:08.994413 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:08.994363    2579 scope.go:117] "RemoveContainer" containerID="6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"
Sep 15 21:04:09.005832 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.005243    2579 scope.go:117] "RemoveContainer" containerID="6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"
Sep 15 21:04:09.005954 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:04:09.005925    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed\": container with ID starting with 6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed not found: ID does not exist" containerID="6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"
Sep 15 21:04:09.006043 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.005952    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed"} err="failed to get container status \"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed\": rpc error: code = NotFound desc = could not find container \"6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed\": container with ID starting with 6ef908598516d8b673692c56f18c3f29ae43dd57db6d7b43d4919683772e9aed not found: ID does not exist"
Sep 15 21:04:09.008622 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.008157    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:04:09.017436 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.013746    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-65001159-f5795c96b-b2zxk"]
Sep 15 21:04:09.028375 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.028343    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:04:09.032457 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.032432    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-65001159-5cf7d5c88b-hrdm2"]
Sep 15 21:04:09.669549 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.669516    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2275864e-47a7-4bf5-bfec-51d2624d5030" path="/var/lib/kubelet/pods/2275864e-47a7-4bf5-bfec-51d2624d5030/volumes"
Sep 15 21:04:09.669909 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:09.669898    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c92913-b961-4473-ae88-695c74d56249" path="/var/lib/kubelet/pods/c8c92913-b961-4473-ae88-695c74d56249/volumes"
Sep 15 21:04:34.475647 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:04:34.475617    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:05:02.171023 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:02.170975    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:05:04.020601 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.020557    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:05:04.021027 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.020981    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2275864e-47a7-4bf5-bfec-51d2624d5030" containerName="payload-pre-processing"
Sep 15 21:05:04.021027 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.020992    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2275864e-47a7-4bf5-bfec-51d2624d5030" containerName="payload-pre-processing"
Sep 15 21:05:04.021117 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021003    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c92913-b961-4473-ae88-695c74d56249" containerName="maas-api"
Sep 15 21:05:04.021117 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c92913-b961-4473-ae88-695c74d56249" containerName="maas-api"
Sep 15 21:05:04.021117 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021049    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0cbbdc9-8d68-4827-b3b9-512648b86028" containerName="main"
Sep 15 21:05:04.021117 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021055    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0cbbdc9-8d68-4827-b3b9-512648b86028" containerName="main"
Sep 15 21:05:04.021267 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021126    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0cbbdc9-8d68-4827-b3b9-512648b86028" containerName="main"
Sep 15 21:05:04.021267 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021137    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2275864e-47a7-4bf5-bfec-51d2624d5030" containerName="payload-pre-processing"
Sep 15 21:05:04.021267 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.021143    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c92913-b961-4473-ae88-695c74d56249" containerName="maas-api"
Sep 15 21:05:04.025517 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.025503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.030131 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.030111    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"auth-test-model-728877-kserve-self-signed-certs\""
Sep 15 21:05:04.031632 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.031613    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"openshift-service-ca.crt\""
Sep 15 21:05:04.031736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.031717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"kube-root-ca.crt\""
Sep 15 21:05:04.031736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.031725    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-728877\"/\"default-dockercfg-dm9lj\""
Sep 15 21:05:04.041866 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.041838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:05:04.140099 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.140283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140117    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.140283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.140283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140254    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.140283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140279    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hts9d\" (UniqueName: \"kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.140428 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.140329    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241119 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hts9d\" (UniqueName: \"kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241287    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241540 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241676 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241743 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241655    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.241743 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.241696    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.243641 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.243620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.243864 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.243843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.250769 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.250748    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hts9d\" (UniqueName: \"kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d\") pod \"auth-test-model-728877-kserve-5b4957d6dc-w6pvq\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") " pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.335264 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.335169    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:04.619429 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:04.619407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:05:04.622155 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:05:04.622128    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ee3f14_f61b_40fc_ae7a_67582e8c7b69.slice/crio-75e6cc8195f5d319fc89aa667d33a37696d4185507ba62b25c4f43dc03bd236f WatchSource:0}: Error finding container 75e6cc8195f5d319fc89aa667d33a37696d4185507ba62b25c4f43dc03bd236f: Status 404 returned error can't find the container with id 75e6cc8195f5d319fc89aa667d33a37696d4185507ba62b25c4f43dc03bd236f
Sep 15 21:05:05.271264 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:05.271230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" event={"ID":"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69","Type":"ContainerStarted","Data":"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"}
Sep 15 21:05:05.271671 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:05.271273    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" event={"ID":"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69","Type":"ContainerStarted","Data":"75e6cc8195f5d319fc89aa667d33a37696d4185507ba62b25c4f43dc03bd236f"}
Sep 15 21:05:05.291987 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:05.291932    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" podStartSLOduration=2.291918279 podStartE2EDuration="2.291918279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:05:05.290472007 +0000 UTC m=+1696.077596035" watchObservedRunningTime="2026-09-15 21:05:05.291918279 +0000 UTC m=+1696.079042307"
Sep 15 21:05:14.335697 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:14.335564    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:14.335697 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:14.335664    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:14.348515 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:14.348491    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:15.342704 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:15.342666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:05:27.891030 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:05:27.887312    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:06:01.573714 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:01.573677    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:06:27.576979 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:27.576941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:06:34.047485 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.047448    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:06:34.051269 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.051248    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.054109 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.054086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-178002-serving-cert\""
Sep 15 21:06:34.064500 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.064471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:06:34.216575 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.216527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbmdr\" (UniqueName: \"kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.216575 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.216580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.216835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.216672    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.216835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.216721    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.317588 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.317481    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbmdr\" (UniqueName: \"kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.317588 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.317528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.317588 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.317581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.317897 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.317613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.318314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.318294    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.320409 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.320378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.320505 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.320378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.327406 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.327373    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbmdr\" (UniqueName: \"kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr\") pod \"maas-api-e2e-api-a-178002-744c99994b-t7dj2\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.362122 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.362081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:34.641846 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.641793    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:06:34.645591 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:06:34.645559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2c9bc1_0c54_4a34_992e_52c4b654ad6c.slice/crio-41671db6aa602a77c13fc1ce4b9a1027129386c1cd77292a2f9b53f402adf4f4 WatchSource:0}: Error finding container 41671db6aa602a77c13fc1ce4b9a1027129386c1cd77292a2f9b53f402adf4f4: Status 404 returned error can't find the container with id 41671db6aa602a77c13fc1ce4b9a1027129386c1cd77292a2f9b53f402adf4f4
Sep 15 21:06:34.647381 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.647361    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:06:34.767443 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.767400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" event={"ID":"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c","Type":"ContainerStarted","Data":"41671db6aa602a77c13fc1ce4b9a1027129386c1cd77292a2f9b53f402adf4f4"}
Sep 15 21:06:34.792143 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.792108    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:06:34.796871 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.796848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:34.799853 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.799831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-178002\""
Sep 15 21:06:34.799981 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.799842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-178002-dockercfg-r64sj\""
Sep 15 21:06:34.806729 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.806705    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:06:34.822596 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.822558    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:06:34.826409 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.826384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:34.836554 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.836525    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:06:34.924405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.924313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gmvn\" (UniqueName: \"kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:34.924405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.924353    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:34.924639 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.924465    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:34.924639 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:34.924506    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztt5x\" (UniqueName: \"kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.025131 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.025072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztt5x\" (UniqueName: \"kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.025328 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.025218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gmvn\" (UniqueName: \"kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.025328 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.025248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.025409 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.025331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.026053 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.026034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.026112 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.026063    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.034064 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.034040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gmvn\" (UniqueName: \"kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn\") pod \"payload-processing-e2e-api-a-178002-7954d74875-9ch7j\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") " pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.034179 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.034042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztt5x\" (UniqueName: \"kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x\") pod \"payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.109877 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.109842    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.138460 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.138430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.385361 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.385334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:06:35.386889 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:06:35.386862    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8388043_8c2d_4170_a510_779bf4b200fb.slice/crio-e42d78301c353bab19711cbb57ba588e2de3403746f92aa163348588c9dd2bf1 WatchSource:0}: Error finding container e42d78301c353bab19711cbb57ba588e2de3403746f92aa163348588c9dd2bf1: Status 404 returned error can't find the container with id e42d78301c353bab19711cbb57ba588e2de3403746f92aa163348588c9dd2bf1
Sep 15 21:06:35.419594 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.419571    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:06:35.422329 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:06:35.422296    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c77026_68eb_4f88_83b4_33bcc74a6773.slice/crio-bcc5091f1ebcb6c477a2a3e761165ea92a52dc46fd94a43ef3529e4b10da8f8f WatchSource:0}: Error finding container bcc5091f1ebcb6c477a2a3e761165ea92a52dc46fd94a43ef3529e4b10da8f8f: Status 404 returned error can't find the container with id bcc5091f1ebcb6c477a2a3e761165ea92a52dc46fd94a43ef3529e4b10da8f8f
Sep 15 21:06:35.775904 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.775796    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" event={"ID":"a8c77026-68eb-4f88-83b4-33bcc74a6773","Type":"ContainerStarted","Data":"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"}
Sep 15 21:06:35.775904 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.775866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" event={"ID":"a8c77026-68eb-4f88-83b4-33bcc74a6773","Type":"ContainerStarted","Data":"bcc5091f1ebcb6c477a2a3e761165ea92a52dc46fd94a43ef3529e4b10da8f8f"}
Sep 15 21:06:35.776176 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.776031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:35.777614 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.777590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" event={"ID":"f8388043-8c2d-4170-a510-779bf4b200fb","Type":"ContainerStarted","Data":"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"}
Sep 15 21:06:35.777723 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.777634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" event={"ID":"f8388043-8c2d-4170-a510-779bf4b200fb","Type":"ContainerStarted","Data":"e42d78301c353bab19711cbb57ba588e2de3403746f92aa163348588c9dd2bf1"}
Sep 15 21:06:35.777786 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.777762    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:35.792950 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.792900    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" podStartSLOduration=1.79288183 podStartE2EDuration="1.79288183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:35.791573592 +0000 UTC m=+1786.578697622" watchObservedRunningTime="2026-09-15 21:06:35.79288183 +0000 UTC m=+1786.580005936"
Sep 15 21:06:35.815448 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:35.815393    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" podStartSLOduration=1.815375021 podStartE2EDuration="1.815375021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:35.812943541 +0000 UTC m=+1786.600067571" watchObservedRunningTime="2026-09-15 21:06:35.815375021 +0000 UTC m=+1786.602499049"
Sep 15 21:06:36.784477 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:36.784441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" event={"ID":"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c","Type":"ContainerStarted","Data":"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"}
Sep 15 21:06:36.784971 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:36.784680    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:36.805667 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:36.805617    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" podStartSLOduration=1.195957448 podStartE2EDuration="2.80560084s" totalImagesPullingTime="1.609643392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:06:36.802787048 +0000 UTC m=+1787.589911078" watchObservedRunningTime="2026-09-15 21:06:36.80560084 +0000 UTC m=+1787.592724867"
Sep 15 21:06:42.795633 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:42.795562    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:06:46.786678 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:46.786649    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:06:46.787084 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:46.786718    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:06:49.825374 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:06:49.825336    2579 scope.go:117] "RemoveContainer" containerID="55b1d11631e84593e2fd939ec9ffc86886bddb6702e82e0a9bc3970a84d53d83"
Sep 15 21:07:10.548685 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.548645    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:07:10.553630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.553607    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.557458 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.557430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14aeb7-dockercfg-zflsd\""
Sep 15 21:07:10.557614 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.557494    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14aeb7\""
Sep 15 21:07:10.565763 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.565740    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:07:10.700545 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.700510    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.700720 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.700594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78k9f\" (UniqueName: \"kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.801440 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.801350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78k9f\" (UniqueName: \"kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.801440 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.801442    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.802132 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.802114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.815152 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.815126    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78k9f\" (UniqueName: \"kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f\") pod \"payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:10.867882 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:10.867852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:11.140639 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:11.140609    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:07:11.144061 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:07:11.144028    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0522acd7_9380_47e5_a7c3_1491252b4abe.slice/crio-857ffbe0ead71870d25e79e1e922ba25f48ee500f7c088c073d315f876d6691f WatchSource:0}: Error finding container 857ffbe0ead71870d25e79e1e922ba25f48ee500f7c088c073d315f876d6691f: Status 404 returned error can't find the container with id 857ffbe0ead71870d25e79e1e922ba25f48ee500f7c088c073d315f876d6691f
Sep 15 21:07:11.978616 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:11.978577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" event={"ID":"0522acd7-9380-47e5-a7c3-1491252b4abe","Type":"ContainerStarted","Data":"3ac279a96eea971250665240ebee79fc0ded874c46735afb8f5795f5935ae0dd"}
Sep 15 21:07:11.979171 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:11.978642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" event={"ID":"0522acd7-9380-47e5-a7c3-1491252b4abe","Type":"ContainerStarted","Data":"857ffbe0ead71870d25e79e1e922ba25f48ee500f7c088c073d315f876d6691f"}
Sep 15 21:07:11.979171 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:11.978741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:11.996424 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:11.996372    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" podStartSLOduration=1.996356086 podStartE2EDuration="1.996356086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:07:11.993865241 +0000 UTC m=+1822.780989271" watchObservedRunningTime="2026-09-15 21:07:11.996356086 +0000 UTC m=+1822.783480115"
Sep 15 21:07:12.669547 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:12.669511    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:07:22.987483 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:22.987447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:07:52.667067 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.667027    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:07:52.752091 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.752050    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:07:52.752302 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.752277    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerName="maas-api" containerID="cri-o://b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756" gracePeriod=30
Sep 15 21:07:52.791164 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.791129    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.62:8443/health\": dial tcp 10.133.0.62:8443: connect: connection refused"
Sep 15 21:07:52.807356 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.807326    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:07:52.807541 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.807514    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" podUID="a8c77026-68eb-4f88-83b4-33bcc74a6773" containerName="payload-processing" containerID="cri-o://5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7" gracePeriod=30
Sep 15 21:07:52.813998 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.813969    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:07:52.814225 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:52.814201    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" podUID="f8388043-8c2d-4170-a510-779bf4b200fb" containerName="payload-pre-processing" containerID="cri-o://7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca" gracePeriod=30
Sep 15 21:07:53.098114 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.098083    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:07:53.119974 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.119948    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:07:53.134003 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.133978    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:07:53.135580 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.135556    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbmdr\" (UniqueName: \"kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr\") pod \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") "
Sep 15 21:07:53.135703 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.135603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca\") pod \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") "
Sep 15 21:07:53.135703 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.135654    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs\") pod \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") "
Sep 15 21:07:53.135703 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.135693    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls\") pod \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\" (UID: \"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c\") "
Sep 15 21:07:53.136070 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.136039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" (UID: "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:07:53.137859 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.137835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" (UID: "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:07:53.137949 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.137896    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" (UID: "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:07:53.138242 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.138218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr" (OuterVolumeSpecName: "kube-api-access-dbmdr") pod "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" (UID: "8d2c9bc1-0c54-4a34-992e-52c4b654ad6c"). InnerVolumeSpecName "kube-api-access-dbmdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:07:53.194784 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.194699    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerID="b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756" exitCode=0
Sep 15 21:07:53.194784 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.194743    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"
Sep 15 21:07:53.195002 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.194795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" event={"ID":"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c","Type":"ContainerDied","Data":"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"}
Sep 15 21:07:53.195002 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.194843    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2" event={"ID":"8d2c9bc1-0c54-4a34-992e-52c4b654ad6c","Type":"ContainerDied","Data":"41671db6aa602a77c13fc1ce4b9a1027129386c1cd77292a2f9b53f402adf4f4"}
Sep 15 21:07:53.195002 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.194865    2579 scope.go:117] "RemoveContainer" containerID="b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"
Sep 15 21:07:53.196288 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.196270    2579 generic.go:356] "Generic (PLEG): container finished" podID="a8c77026-68eb-4f88-83b4-33bcc74a6773" containerID="5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7" exitCode=0
Sep 15 21:07:53.196405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.196316    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"
Sep 15 21:07:53.196405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.196346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" event={"ID":"a8c77026-68eb-4f88-83b4-33bcc74a6773","Type":"ContainerDied","Data":"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"}
Sep 15 21:07:53.196405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.196398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j" event={"ID":"a8c77026-68eb-4f88-83b4-33bcc74a6773","Type":"ContainerDied","Data":"bcc5091f1ebcb6c477a2a3e761165ea92a52dc46fd94a43ef3529e4b10da8f8f"}
Sep 15 21:07:53.197804 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.197782    2579 generic.go:356] "Generic (PLEG): container finished" podID="f8388043-8c2d-4170-a510-779bf4b200fb" containerID="7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca" exitCode=0
Sep 15 21:07:53.197908 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.197823    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"
Sep 15 21:07:53.197908 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.197835    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" event={"ID":"f8388043-8c2d-4170-a510-779bf4b200fb","Type":"ContainerDied","Data":"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"}
Sep 15 21:07:53.197908 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.197887    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll" event={"ID":"f8388043-8c2d-4170-a510-779bf4b200fb","Type":"ContainerDied","Data":"e42d78301c353bab19711cbb57ba588e2de3403746f92aa163348588c9dd2bf1"}
Sep 15 21:07:53.206739 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.206721    2579 scope.go:117] "RemoveContainer" containerID="b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"
Sep 15 21:07:53.207031 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:07:53.206996    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756\": container with ID starting with b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756 not found: ID does not exist" containerID="b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"
Sep 15 21:07:53.207091 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.207040    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756"} err="failed to get container status \"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756\": rpc error: code = NotFound desc = could not find container \"b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756\": container with ID starting with b970512d7a6a0ea4a871557591ed2e7916f4319f3ba650792e84a62119d0b756 not found: ID does not exist"
Sep 15 21:07:53.207091 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.207058    2579 scope.go:117] "RemoveContainer" containerID="5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"
Sep 15 21:07:53.215167 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.215151    2579 scope.go:117] "RemoveContainer" containerID="5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"
Sep 15 21:07:53.215486 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:07:53.215461    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7\": container with ID starting with 5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7 not found: ID does not exist" containerID="5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"
Sep 15 21:07:53.215567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.215496    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7"} err="failed to get container status \"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7\": rpc error: code = NotFound desc = could not find container \"5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7\": container with ID starting with 5550a9122f3b8710cf5a4ab45b7dfb14b3916f9853bdcb06454c0af43c9fbcc7 not found: ID does not exist"
Sep 15 21:07:53.215567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.215519    2579 scope.go:117] "RemoveContainer" containerID="7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"
Sep 15 21:07:53.218574 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.218556    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:07:53.224154 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.224105    2579 scope.go:117] "RemoveContainer" containerID="7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"
Sep 15 21:07:53.224325 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.224306    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-178002-744c99994b-t7dj2"]
Sep 15 21:07:53.224417 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:07:53.224398    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca\": container with ID starting with 7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca not found: ID does not exist" containerID="7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"
Sep 15 21:07:53.224460 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.224425    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca"} err="failed to get container status \"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca\": rpc error: code = NotFound desc = could not find container \"7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca\": container with ID starting with 7ce6fd7a1b5924dee1b954387c08ed95da88d74f6b562d6d497b8fca15c856ca not found: ID does not exist"
Sep 15 21:07:53.237000 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.236981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztt5x\" (UniqueName: \"kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x\") pod \"f8388043-8c2d-4170-a510-779bf4b200fb\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") "
Sep 15 21:07:53.237125 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume\") pod \"a8c77026-68eb-4f88-83b4-33bcc74a6773\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") "
Sep 15 21:07:53.237184 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume\") pod \"f8388043-8c2d-4170-a510-779bf4b200fb\" (UID: \"f8388043-8c2d-4170-a510-779bf4b200fb\") "
Sep 15 21:07:53.237184 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237174    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gmvn\" (UniqueName: \"kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn\") pod \"a8c77026-68eb-4f88-83b4-33bcc74a6773\" (UID: \"a8c77026-68eb-4f88-83b4-33bcc74a6773\") "
Sep 15 21:07:53.237482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8c77026-68eb-4f88-83b4-33bcc74a6773" (UID: "a8c77026-68eb-4f88-83b4-33bcc74a6773"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:07:53.237567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237461    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbmdr\" (UniqueName: \"kubernetes.io/projected/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-kube-api-access-dbmdr\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.237567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237511    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-openshift-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.237567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237529    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.237567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8388043-8c2d-4170-a510-779bf4b200fb" (UID: "f8388043-8c2d-4170-a510-779bf4b200fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:07:53.237567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.237542    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c-maas-api-tls\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.239778 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.239758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x" (OuterVolumeSpecName: "kube-api-access-ztt5x") pod "f8388043-8c2d-4170-a510-779bf4b200fb" (UID: "f8388043-8c2d-4170-a510-779bf4b200fb"). InnerVolumeSpecName "kube-api-access-ztt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:07:53.240175 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.240155    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn" (OuterVolumeSpecName: "kube-api-access-7gmvn") pod "a8c77026-68eb-4f88-83b4-33bcc74a6773" (UID: "a8c77026-68eb-4f88-83b4-33bcc74a6773"). InnerVolumeSpecName "kube-api-access-7gmvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:07:53.338852 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.338811    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8388043-8c2d-4170-a510-779bf4b200fb-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.338852 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.338846    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gmvn\" (UniqueName: \"kubernetes.io/projected/a8c77026-68eb-4f88-83b4-33bcc74a6773-kube-api-access-7gmvn\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.338852 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.338855    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztt5x\" (UniqueName: \"kubernetes.io/projected/f8388043-8c2d-4170-a510-779bf4b200fb-kube-api-access-ztt5x\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.338852 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.338864    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8c77026-68eb-4f88-83b4-33bcc74a6773-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:07:53.536464 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.536435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:07:53.546845 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.546815    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-178002-6464bf55fd-7dxll"]
Sep 15 21:07:53.561982 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.561956    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:07:53.572960 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.572932    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-178002-7954d74875-9ch7j"]
Sep 15 21:07:53.668906 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.668875    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" path="/var/lib/kubelet/pods/8d2c9bc1-0c54-4a34-992e-52c4b654ad6c/volumes"
Sep 15 21:07:53.669317 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.669282    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8c77026-68eb-4f88-83b4-33bcc74a6773" path="/var/lib/kubelet/pods/a8c77026-68eb-4f88-83b4-33bcc74a6773/volumes"
Sep 15 21:07:53.669596 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:07:53.669584    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8388043-8c2d-4170-a510-779bf4b200fb" path="/var/lib/kubelet/pods/f8388043-8c2d-4170-a510-779bf4b200fb/volumes"
Sep 15 21:08:05.520678 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:05.520637    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:08:05.566328 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:05.566297    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 21:08:15.935300 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:15.935262    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 21:08:15.935700 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:15.935536    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" podUID="d16c874b-803e-4e91-97dc-de9dc10a7d98" containerName="payload-processing" containerID="cri-o://965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262" gracePeriod=30
Sep 15 21:08:16.127334 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.127299    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:08:16.127584 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.127557    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" podUID="0522acd7-9380-47e5-a7c3-1491252b4abe" containerName="payload-pre-processing" containerID="cri-o://3ac279a96eea971250665240ebee79fc0ded874c46735afb8f5795f5935ae0dd" gracePeriod=30
Sep 15 21:08:16.255986 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.255961    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 21:08:16.331985 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.331950    2579 generic.go:356] "Generic (PLEG): container finished" podID="d16c874b-803e-4e91-97dc-de9dc10a7d98" containerID="965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262" exitCode=0
Sep 15 21:08:16.332548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.332032    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"
Sep 15 21:08:16.332548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.332045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" event={"ID":"d16c874b-803e-4e91-97dc-de9dc10a7d98","Type":"ContainerDied","Data":"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"}
Sep 15 21:08:16.332548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.332153    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc" event={"ID":"d16c874b-803e-4e91-97dc-de9dc10a7d98","Type":"ContainerDied","Data":"31ba2f7afc0910eafe000d4d6c139d5856f5c0de183dbf9ef1bbf031d22e2e7f"}
Sep 15 21:08:16.332548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.332199    2579 scope.go:117] "RemoveContainer" containerID="965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"
Sep 15 21:08:16.334685 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.334658    2579 generic.go:356] "Generic (PLEG): container finished" podID="0522acd7-9380-47e5-a7c3-1491252b4abe" containerID="3ac279a96eea971250665240ebee79fc0ded874c46735afb8f5795f5935ae0dd" exitCode=0
Sep 15 21:08:16.334813 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.334709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" event={"ID":"0522acd7-9380-47e5-a7c3-1491252b4abe","Type":"ContainerDied","Data":"3ac279a96eea971250665240ebee79fc0ded874c46735afb8f5795f5935ae0dd"}
Sep 15 21:08:16.342992 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.342962    2579 scope.go:117] "RemoveContainer" containerID="965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"
Sep 15 21:08:16.343402 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:16.343372    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262\": container with ID starting with 965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262 not found: ID does not exist" containerID="965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"
Sep 15 21:08:16.343503 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.343413    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262"} err="failed to get container status \"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262\": rpc error: code = NotFound desc = could not find container \"965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262\": container with ID starting with 965a469626de6eed770f0f08b85d9edf7b13000680b34dd54a5b0a06d86bb262 not found: ID does not exist"
Sep 15 21:08:16.369799 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.369769    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97rrm\" (UniqueName: \"kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm\") pod \"d16c874b-803e-4e91-97dc-de9dc10a7d98\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") "
Sep 15 21:08:16.369983 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.369888    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume\") pod \"d16c874b-803e-4e91-97dc-de9dc10a7d98\" (UID: \"d16c874b-803e-4e91-97dc-de9dc10a7d98\") "
Sep 15 21:08:16.370374 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.370345    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d16c874b-803e-4e91-97dc-de9dc10a7d98" (UID: "d16c874b-803e-4e91-97dc-de9dc10a7d98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:08:16.372337 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.372305    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm" (OuterVolumeSpecName: "kube-api-access-97rrm") pod "d16c874b-803e-4e91-97dc-de9dc10a7d98" (UID: "d16c874b-803e-4e91-97dc-de9dc10a7d98"). InnerVolumeSpecName "kube-api-access-97rrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:16.387548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.387522    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:08:16.471933 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.471831    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d16c874b-803e-4e91-97dc-de9dc10a7d98-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:16.471933 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.471868    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97rrm\" (UniqueName: \"kubernetes.io/projected/d16c874b-803e-4e91-97dc-de9dc10a7d98-kube-api-access-97rrm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:16.572251 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.572196    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78k9f\" (UniqueName: \"kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f\") pod \"0522acd7-9380-47e5-a7c3-1491252b4abe\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") "
Sep 15 21:08:16.572462 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.572372    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume\") pod \"0522acd7-9380-47e5-a7c3-1491252b4abe\" (UID: \"0522acd7-9380-47e5-a7c3-1491252b4abe\") "
Sep 15 21:08:16.572736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.572707    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0522acd7-9380-47e5-a7c3-1491252b4abe" (UID: "0522acd7-9380-47e5-a7c3-1491252b4abe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:08:16.574655 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.574631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f" (OuterVolumeSpecName: "kube-api-access-78k9f") pod "0522acd7-9380-47e5-a7c3-1491252b4abe" (UID: "0522acd7-9380-47e5-a7c3-1491252b4abe"). InnerVolumeSpecName "kube-api-access-78k9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:16.655526 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.655494    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 21:08:16.660227 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.660201    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-3f1c93-68548778b9-gt4zc"]
Sep 15 21:08:16.673919 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.673890    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0522acd7-9380-47e5-a7c3-1491252b4abe-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:16.673919 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:16.673913    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78k9f\" (UniqueName: \"kubernetes.io/projected/0522acd7-9380-47e5-a7c3-1491252b4abe-kube-api-access-78k9f\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:17.341311 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.341226    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"
Sep 15 21:08:17.341311 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.341261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z" event={"ID":"0522acd7-9380-47e5-a7c3-1491252b4abe","Type":"ContainerDied","Data":"857ffbe0ead71870d25e79e1e922ba25f48ee500f7c088c073d315f876d6691f"}
Sep 15 21:08:17.341806 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.341340    2579 scope.go:117] "RemoveContainer" containerID="3ac279a96eea971250665240ebee79fc0ded874c46735afb8f5795f5935ae0dd"
Sep 15 21:08:17.373873 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.372411    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:08:17.373873 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.372475    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-14aeb7-58664559cd-ptq7z"]
Sep 15 21:08:17.669000 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.668965    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0522acd7-9380-47e5-a7c3-1491252b4abe" path="/var/lib/kubelet/pods/0522acd7-9380-47e5-a7c3-1491252b4abe/volumes"
Sep 15 21:08:17.669372 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:17.669358    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d16c874b-803e-4e91-97dc-de9dc10a7d98" path="/var/lib/kubelet/pods/d16c874b-803e-4e91-97dc-de9dc10a7d98/volumes"
Sep 15 21:08:20.540949 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.540907    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" podUID="f9920da0-0fd3-4003-8604-ebd479f34c2d" containerName="main" containerID="cri-o://95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a" gracePeriod=45
Sep 15 21:08:20.586105 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.586056    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" podUID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" containerName="main" containerID="cri-o://cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77" gracePeriod=45
Sep 15 21:08:20.853000 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.852965    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:08:20.866283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.866220    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 21:08:20.915979 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.915942    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.915979 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.915976    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.916228 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv27q\" (UniqueName: \"kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.916228 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916057    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916228 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916108    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.916228 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916228 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916209    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916234    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916257    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fv8w\" (UniqueName: \"kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916281    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916309    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs\") pod \"f9920da0-0fd3-4003-8604-ebd479f34c2d\" (UID: \"f9920da0-0fd3-4003-8604-ebd479f34c2d\") "
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916300    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home" (OuterVolumeSpecName: "home") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.916482 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916348    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir\") pod \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\" (UID: \"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761\") "
Sep 15 21:08:20.916847 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home" (OuterVolumeSpecName: "home") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.916847 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916703    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:20.916847 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916721    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:20.916995 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache" (OuterVolumeSpecName: "model-cache") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.916995 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.916917    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.917151 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.917078    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.919350 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919323    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm" (OuterVolumeSpecName: "dshm") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.919350 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919320    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w" (OuterVolumeSpecName: "kube-api-access-8fv8w") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "kube-api-access-8fv8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:20.919530 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm" (OuterVolumeSpecName: "dshm") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:08:20.919530 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:08:20.919640 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" (UID: "4c84908c-bdf9-4ef6-9e28-5f2f8ce15761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:08:20.919860 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:20.919836    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q" (OuterVolumeSpecName: "kube-api-access-cv27q") pod "f9920da0-0fd3-4003-8604-ebd479f34c2d" (UID: "f9920da0-0fd3-4003-8604-ebd479f34c2d"). InnerVolumeSpecName "kube-api-access-cv27q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:21.017608 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017571    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017623    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017636    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fv8w\" (UniqueName: \"kubernetes.io/projected/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-kube-api-access-8fv8w\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017644    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017672    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9920da0-0fd3-4003-8604-ebd479f34c2d-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017680    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017688    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv27q\" (UniqueName: \"kubernetes.io/projected/f9920da0-0fd3-4003-8604-ebd479f34c2d-kube-api-access-cv27q\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017701    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017711    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9920da0-0fd3-4003-8604-ebd479f34c2d-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.017777 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.017719    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:21.366203 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.366171    2579 generic.go:356] "Generic (PLEG): container finished" podID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" containerID="cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77" exitCode=0
Sep 15 21:08:21.366378 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.366212    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"
Sep 15 21:08:21.366378 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.366265    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" event={"ID":"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761","Type":"ContainerDied","Data":"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"}
Sep 15 21:08:21.366378 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.366308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v" event={"ID":"4c84908c-bdf9-4ef6-9e28-5f2f8ce15761","Type":"ContainerDied","Data":"a2ea87b6f9967d10b2a6bf4b225f81ac446ae024b453a6f6683b5a3b114216d6"}
Sep 15 21:08:21.366378 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.366331    2579 scope.go:117] "RemoveContainer" containerID="cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"
Sep 15 21:08:21.367911 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.367889    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"
Sep 15 21:08:21.368072 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.367843    2579 generic.go:356] "Generic (PLEG): container finished" podID="f9920da0-0fd3-4003-8604-ebd479f34c2d" containerID="95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a" exitCode=0
Sep 15 21:08:21.368072 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.367945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" event={"ID":"f9920da0-0fd3-4003-8604-ebd479f34c2d","Type":"ContainerDied","Data":"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"}
Sep 15 21:08:21.368180 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.368085    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms" event={"ID":"f9920da0-0fd3-4003-8604-ebd479f34c2d","Type":"ContainerDied","Data":"68a3a3eecbcde83852faa2b8a325dc28d2a892b106911d75153898ea82fadc8c"}
Sep 15 21:08:21.375668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.375652    2579 scope.go:117] "RemoveContainer" containerID="cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"
Sep 15 21:08:21.375963 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:21.375945    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77\": container with ID starting with cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77 not found: ID does not exist" containerID="cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"
Sep 15 21:08:21.376100 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.375972    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77"} err="failed to get container status \"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77\": rpc error: code = NotFound desc = could not find container \"cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77\": container with ID starting with cfe4c5d3d10106198f0cfef408298051623e7c4bc78806de73eb5078a9881c77 not found: ID does not exist"
Sep 15 21:08:21.376100 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.375989    2579 scope.go:117] "RemoveContainer" containerID="95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"
Sep 15 21:08:21.384249 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.384231    2579 scope.go:117] "RemoveContainer" containerID="95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"
Sep 15 21:08:21.384551 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:21.384533    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a\": container with ID starting with 95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a not found: ID does not exist" containerID="95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"
Sep 15 21:08:21.384625 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.384557    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a"} err="failed to get container status \"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a\": rpc error: code = NotFound desc = could not find container \"95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a\": container with ID starting with 95dfba053aad15b3af416b13b52b5e1a6cd76c6c53c4b3ccec0b1f8b0df25f7a not found: ID does not exist"
Sep 15 21:08:21.401638 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.401612    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:08:21.403918 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.403897    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/distinct-2-w0-3f1c93-kserve-79f6bc9f4f-8zrms"]
Sep 15 21:08:21.416186 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.416158    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 21:08:21.420720 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.420692    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3f1c93/simulator-w0-3f1c93-kserve-cfc4c8966-g7m7v"]
Sep 15 21:08:21.669315 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.669276    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" path="/var/lib/kubelet/pods/4c84908c-bdf9-4ef6-9e28-5f2f8ce15761/volumes"
Sep 15 21:08:21.669736 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:21.669689    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9920da0-0fd3-4003-8604-ebd479f34c2d" path="/var/lib/kubelet/pods/f9920da0-0fd3-4003-8604-ebd479f34c2d/volumes"
Sep 15 21:08:35.982567 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:35.982532    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 21:08:35.983066 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:35.982845    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" podUID="59e13b92-3837-4acd-a5ec-e4edf0776714" containerName="payload-processing" containerID="cri-o://0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39" gracePeriod=30
Sep 15 21:08:35.990684 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:35.990280    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 21:08:35.990684 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:35.990563    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" podUID="cb8a07c2-4b70-4f15-8936-2642627d8363" containerName="payload-pre-processing" containerID="cri-o://588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d" gracePeriod=30
Sep 15 21:08:36.257702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.257674    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 21:08:36.289586 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.289554    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 21:08:36.364951 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.364920    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmvvw\" (UniqueName: \"kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw\") pod \"59e13b92-3837-4acd-a5ec-e4edf0776714\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") "
Sep 15 21:08:36.365175 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.365044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume\") pod \"59e13b92-3837-4acd-a5ec-e4edf0776714\" (UID: \"59e13b92-3837-4acd-a5ec-e4edf0776714\") "
Sep 15 21:08:36.365175 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.365089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj9tb\" (UniqueName: \"kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb\") pod \"cb8a07c2-4b70-4f15-8936-2642627d8363\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") "
Sep 15 21:08:36.365175 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.365112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume\") pod \"cb8a07c2-4b70-4f15-8936-2642627d8363\" (UID: \"cb8a07c2-4b70-4f15-8936-2642627d8363\") "
Sep 15 21:08:36.365489 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.365456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "59e13b92-3837-4acd-a5ec-e4edf0776714" (UID: "59e13b92-3837-4acd-a5ec-e4edf0776714"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:08:36.365603 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.365518    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb8a07c2-4b70-4f15-8936-2642627d8363" (UID: "cb8a07c2-4b70-4f15-8936-2642627d8363"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:08:36.367319 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.367296    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw" (OuterVolumeSpecName: "kube-api-access-pmvvw") pod "59e13b92-3837-4acd-a5ec-e4edf0776714" (UID: "59e13b92-3837-4acd-a5ec-e4edf0776714"). InnerVolumeSpecName "kube-api-access-pmvvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:36.367319 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.367312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb" (OuterVolumeSpecName: "kube-api-access-zj9tb") pod "cb8a07c2-4b70-4f15-8936-2642627d8363" (UID: "cb8a07c2-4b70-4f15-8936-2642627d8363"). InnerVolumeSpecName "kube-api-access-zj9tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:36.450999 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.450966    2579 generic.go:356] "Generic (PLEG): container finished" podID="cb8a07c2-4b70-4f15-8936-2642627d8363" containerID="588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d" exitCode=0
Sep 15 21:08:36.451232 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.451044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" event={"ID":"cb8a07c2-4b70-4f15-8936-2642627d8363","Type":"ContainerDied","Data":"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"}
Sep 15 21:08:36.451232 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.451057    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"
Sep 15 21:08:36.451232 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.451143    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt" event={"ID":"cb8a07c2-4b70-4f15-8936-2642627d8363","Type":"ContainerDied","Data":"d074f4f6f6e55f3ce3d10028a4985a5b85e17488674753186f72edaad600ec64"}
Sep 15 21:08:36.451232 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.451179    2579 scope.go:117] "RemoveContainer" containerID="588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"
Sep 15 21:08:36.452698 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.452681    2579 generic.go:356] "Generic (PLEG): container finished" podID="59e13b92-3837-4acd-a5ec-e4edf0776714" containerID="0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39" exitCode=0
Sep 15 21:08:36.452773 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.452724    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"
Sep 15 21:08:36.452843 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.452807    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" event={"ID":"59e13b92-3837-4acd-a5ec-e4edf0776714","Type":"ContainerDied","Data":"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"}
Sep 15 21:08:36.452905 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.452881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2" event={"ID":"59e13b92-3837-4acd-a5ec-e4edf0776714","Type":"ContainerDied","Data":"e051d781b2e4be032525e6b686706ad9a1a92813ca7739b59247d3977af70880"}
Sep 15 21:08:36.461906 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.461885    2579 scope.go:117] "RemoveContainer" containerID="588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"
Sep 15 21:08:36.462283 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:36.462255    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d\": container with ID starting with 588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d not found: ID does not exist" containerID="588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"
Sep 15 21:08:36.462377 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.462296    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d"} err="failed to get container status \"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d\": rpc error: code = NotFound desc = could not find container \"588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d\": container with ID starting with 588ec001f38f227ab7e0b27c2e61f7663e829db307c58db9d2a0c08180d7d78d not found: ID does not exist"
Sep 15 21:08:36.462377 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.462315    2579 scope.go:117] "RemoveContainer" containerID="0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"
Sep 15 21:08:36.466308 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.466284    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59e13b92-3837-4acd-a5ec-e4edf0776714-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:36.466405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.466314    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj9tb\" (UniqueName: \"kubernetes.io/projected/cb8a07c2-4b70-4f15-8936-2642627d8363-kube-api-access-zj9tb\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:36.466405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.466335    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb8a07c2-4b70-4f15-8936-2642627d8363-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:36.466405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.466353    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmvvw\" (UniqueName: \"kubernetes.io/projected/59e13b92-3837-4acd-a5ec-e4edf0776714-kube-api-access-pmvvw\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:36.470708 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.470689    2579 scope.go:117] "RemoveContainer" containerID="0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"
Sep 15 21:08:36.471087 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:36.471022    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39\": container with ID starting with 0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39 not found: ID does not exist" containerID="0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"
Sep 15 21:08:36.471183 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.471121    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39"} err="failed to get container status \"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39\": rpc error: code = NotFound desc = could not find container \"0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39\": container with ID starting with 0bb17cc4a8129ed3e93dcfae08cb9f09cd9f66ca9e5f9e98864f621474e00f39 not found: ID does not exist"
Sep 15 21:08:36.484635 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.484581    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 21:08:36.488484 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.488465    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-de4215-985b49878-782kt"]
Sep 15 21:08:36.501655 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.501633    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 21:08:36.505351 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:36.505332    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-de4215-78bd9bf9f9-9vfr2"]
Sep 15 21:08:37.088548 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:37.088519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:08:37.668243 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:37.668201    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59e13b92-3837-4acd-a5ec-e4edf0776714" path="/var/lib/kubelet/pods/59e13b92-3837-4acd-a5ec-e4edf0776714/volumes"
Sep 15 21:08:37.668693 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:37.668671    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8a07c2-4b70-4f15-8936-2642627d8363" path="/var/lib/kubelet/pods/cb8a07c2-4b70-4f15-8936-2642627d8363/volumes"
Sep 15 21:08:54.625584 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:54.625548    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:08:54.626035 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:54.625765    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" podUID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" containerName="payload-pre-processing" containerID="cri-o://fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c" gracePeriod=30
Sep 15 21:08:54.891929 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:54.891903    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:08:55.049277 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.049241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlbmq\" (UniqueName: \"kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq\") pod \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") "
Sep 15 21:08:55.049455 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.049377    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume\") pod \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\" (UID: \"cfe32c0f-f416-4a69-83bd-4d1958bf63ea\") "
Sep 15 21:08:55.049753 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.049728    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfe32c0f-f416-4a69-83bd-4d1958bf63ea" (UID: "cfe32c0f-f416-4a69-83bd-4d1958bf63ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:08:55.051472 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.051451    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq" (OuterVolumeSpecName: "kube-api-access-xlbmq") pod "cfe32c0f-f416-4a69-83bd-4d1958bf63ea" (UID: "cfe32c0f-f416-4a69-83bd-4d1958bf63ea"). InnerVolumeSpecName "kube-api-access-xlbmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:08:55.151002 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.150923    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:55.151002 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.150949    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlbmq\" (UniqueName: \"kubernetes.io/projected/cfe32c0f-f416-4a69-83bd-4d1958bf63ea-kube-api-access-xlbmq\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:08:55.550711 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.550622    2579 generic.go:356] "Generic (PLEG): container finished" podID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" containerID="fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c" exitCode=0
Sep 15 21:08:55.550711 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.550677    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"
Sep 15 21:08:55.550902 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.550708    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" event={"ID":"cfe32c0f-f416-4a69-83bd-4d1958bf63ea","Type":"ContainerDied","Data":"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"}
Sep 15 21:08:55.550902 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.550760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh" event={"ID":"cfe32c0f-f416-4a69-83bd-4d1958bf63ea","Type":"ContainerDied","Data":"0795c684f5fe72e917d104085f7d3064cda3095a6fbb4bfa2c8d7d4ee79c72d9"}
Sep 15 21:08:55.550902 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.550783    2579 scope.go:117] "RemoveContainer" containerID="fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"
Sep 15 21:08:55.560029 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.559989    2579 scope.go:117] "RemoveContainer" containerID="fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"
Sep 15 21:08:55.560346 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:08:55.560329    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c\": container with ID starting with fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c not found: ID does not exist" containerID="fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"
Sep 15 21:08:55.560411 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.560353    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c"} err="failed to get container status \"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c\": rpc error: code = NotFound desc = could not find container \"fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c\": container with ID starting with fdb0f030932bf44c30823f9c1ffd698ceaa5a06ce922990251df26889870164c not found: ID does not exist"
Sep 15 21:08:55.579748 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.579717    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:08:55.587730 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.587705    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-3825fc-5954fbbb78-jjbrh"]
Sep 15 21:08:55.667885 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:08:55.667852    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" path="/var/lib/kubelet/pods/cfe32c0f-f416-4a69-83bd-4d1958bf63ea/volumes"
Sep 15 21:09:01.339551 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:01.339516    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:09:22.883388 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:22.883349    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:09:30.254063 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:30.254026    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:09:30.782063 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:30.782023    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:09:42.352843 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.352760    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353142    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerName="maas-api"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353154    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerName="maas-api"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353165    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" containerName="payload-pre-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353171    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" containerName="payload-pre-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353178    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8c77026-68eb-4f88-83b4-33bcc74a6773" containerName="payload-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353184    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8c77026-68eb-4f88-83b4-33bcc74a6773" containerName="payload-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353197    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9920da0-0fd3-4003-8604-ebd479f34c2d" containerName="main"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353203    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9920da0-0fd3-4003-8604-ebd479f34c2d" containerName="main"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353213    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8388043-8c2d-4170-a510-779bf4b200fb" containerName="payload-pre-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353218    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8388043-8c2d-4170-a510-779bf4b200fb" containerName="payload-pre-processing"
Sep 15 21:09:42.353222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353225    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0522acd7-9380-47e5-a7c3-1491252b4abe" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353230    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0522acd7-9380-47e5-a7c3-1491252b4abe" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353237    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" containerName="main"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353241    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" containerName="main"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353249    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d16c874b-803e-4e91-97dc-de9dc10a7d98" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353253    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d16c874b-803e-4e91-97dc-de9dc10a7d98" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353260    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8a07c2-4b70-4f15-8936-2642627d8363" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353266    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8a07c2-4b70-4f15-8936-2642627d8363" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353271    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59e13b92-3837-4acd-a5ec-e4edf0776714" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353275    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="59e13b92-3837-4acd-a5ec-e4edf0776714" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353335    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8a07c2-4b70-4f15-8936-2642627d8363" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353348    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8c77026-68eb-4f88-83b4-33bcc74a6773" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353357    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfe32c0f-f416-4a69-83bd-4d1958bf63ea" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353366    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="59e13b92-3837-4acd-a5ec-e4edf0776714" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353373    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d2c9bc1-0c54-4a34-992e-52c4b654ad6c" containerName="maas-api"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353381    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0522acd7-9380-47e5-a7c3-1491252b4abe" containerName="payload-pre-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353390    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d16c874b-803e-4e91-97dc-de9dc10a7d98" containerName="payload-processing"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353400    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9920da0-0fd3-4003-8604-ebd479f34c2d" containerName="main"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353409    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c84908c-bdf9-4ef6-9e28-5f2f8ce15761" containerName="main"
Sep 15 21:09:42.353740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.353418    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8388043-8c2d-4170-a510-779bf4b200fb" containerName="payload-pre-processing"
Sep 15 21:09:42.356849 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.356826    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.360710 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.360685    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"kube-root-ca.crt\""
Sep 15 21:09:42.360857 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.360729    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"sub-test-model-280fb4-kserve-self-signed-certs\""
Sep 15 21:09:42.360857 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.360752    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"openshift-service-ca.crt\""
Sep 15 21:09:42.360857 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.360789    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-280fb4\"/\"default-dockercfg-m9ttf\""
Sep 15 21:09:42.366890 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.366865    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:09:42.470230 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.470197    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:09:42.483970 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.483941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.484111 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.483975    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.484111 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.483994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn52\" (UniqueName: \"kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.484111 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.484050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.484111 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.484076    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.484267 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.484173    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585411 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585411 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585631 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn52\" (UniqueName: \"kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585631 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585631 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585766 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585735    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585851 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.585914 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.585884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.586140 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.586107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.587954 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.587931    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.588207 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.588190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.593926 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.593906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn52\" (UniqueName: \"kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52\") pod \"sub-test-model-280fb4-kserve-9cdb898dc-8xn5w\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") " pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.666869 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.666833    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:42.939741 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:42.939716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:09:42.941927 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:09:42.941893    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f580886_725f_4254_a248_c6505abbf8c6.slice/crio-ff17441aaa919fb99da821ccca5539310dab7400870bd091da71928063f6edeb WatchSource:0}: Error finding container ff17441aaa919fb99da821ccca5539310dab7400870bd091da71928063f6edeb: Status 404 returned error can't find the container with id ff17441aaa919fb99da821ccca5539310dab7400870bd091da71928063f6edeb
Sep 15 21:09:43.786052 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:43.785999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" event={"ID":"3f580886-725f-4254-a248-c6505abbf8c6","Type":"ContainerStarted","Data":"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"}
Sep 15 21:09:43.786052 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:43.786058    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" event={"ID":"3f580886-725f-4254-a248-c6505abbf8c6","Type":"ContainerStarted","Data":"ff17441aaa919fb99da821ccca5539310dab7400870bd091da71928063f6edeb"}
Sep 15 21:09:43.803087 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:43.803036    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" podStartSLOduration=1.8029911790000002 podStartE2EDuration="1.802991179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:09:43.80185624 +0000 UTC m=+1974.588980268" watchObservedRunningTime="2026-09-15 21:09:43.802991179 +0000 UTC m=+1974.590115208"
Sep 15 21:09:45.275258 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.275196    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerName="main" containerID="cri-o://36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b" gracePeriod=45
Sep 15 21:09:45.332407 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.332371    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": dial tcp 10.133.0.61:8000: connect: connection refused"
Sep 15 21:09:45.371519 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.371468    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 21:09:45.371800 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.371771    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" podUID="46a81999-7bf8-43ff-b62c-8d43046805e2" containerName="maas-api" containerID="cri-o://f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69" gracePeriod=30
Sep 15 21:09:45.430653 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.430559    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 21:09:45.431222 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.431189    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" podUID="74366d23-b564-4e72-b80e-694bd75ab64b" containerName="payload-pre-processing" containerID="cri-o://bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6" gracePeriod=30
Sep 15 21:09:45.562322 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.562287    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:09:45.693333 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.693306    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 21:09:45.705419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.705389    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 21:09:45.714239 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714270    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714290    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714368    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714392    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714659 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hts9d\" (UniqueName: \"kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d\") pod \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\" (UID: \"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69\") "
Sep 15 21:09:45.714853 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:09:45.715154 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714849    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:09:45.715154 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.714887    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home" (OuterVolumeSpecName: "home") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:09:45.717254 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.717228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:09:45.717376 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.717301    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm" (OuterVolumeSpecName: "dshm") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:09:45.717597 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.717571    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d" (OuterVolumeSpecName: "kube-api-access-hts9d") pod "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" (UID: "d5ee3f14-f61b-40fc-ae7a-67582e8c7b69"). InnerVolumeSpecName "kube-api-access-hts9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:09:45.796820 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.796731    2579 generic.go:356] "Generic (PLEG): container finished" podID="46a81999-7bf8-43ff-b62c-8d43046805e2" containerID="f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69" exitCode=0
Sep 15 21:09:45.796820 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.796782    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"
Sep 15 21:09:45.797051 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.796814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" event={"ID":"46a81999-7bf8-43ff-b62c-8d43046805e2","Type":"ContainerDied","Data":"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"}
Sep 15 21:09:45.797051 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.796881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj" event={"ID":"46a81999-7bf8-43ff-b62c-8d43046805e2","Type":"ContainerDied","Data":"d16658a575cda8ce39edd369736a4af76c049c11fa83686b26a3bc06535f6d4d"}
Sep 15 21:09:45.797051 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.796917    2579 scope.go:117] "RemoveContainer" containerID="f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"
Sep 15 21:09:45.798224 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.798205    2579 generic.go:356] "Generic (PLEG): container finished" podID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerID="36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b" exitCode=0
Sep 15 21:09:45.798326 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.798245    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"
Sep 15 21:09:45.798326 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.798252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" event={"ID":"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69","Type":"ContainerDied","Data":"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"}
Sep 15 21:09:45.798326 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.798288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq" event={"ID":"d5ee3f14-f61b-40fc-ae7a-67582e8c7b69","Type":"ContainerDied","Data":"75e6cc8195f5d319fc89aa667d33a37696d4185507ba62b25c4f43dc03bd236f"}
Sep 15 21:09:45.799677 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.799652    2579 generic.go:356] "Generic (PLEG): container finished" podID="74366d23-b564-4e72-b80e-694bd75ab64b" containerID="bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6" exitCode=0
Sep 15 21:09:45.799885 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.799695    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"
Sep 15 21:09:45.799885 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.799696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" event={"ID":"74366d23-b564-4e72-b80e-694bd75ab64b","Type":"ContainerDied","Data":"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"}
Sep 15 21:09:45.799885 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.799736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6" event={"ID":"74366d23-b564-4e72-b80e-694bd75ab64b","Type":"ContainerDied","Data":"b5c828d1cc54cd8c3b82919b34368564923a9bc244ae095eede5d67641121538"}
Sep 15 21:09:45.808075 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.806959    2579 scope.go:117] "RemoveContainer" containerID="f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"
Sep 15 21:09:45.808485 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:09:45.808457    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69\": container with ID starting with f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69 not found: ID does not exist" containerID="f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"
Sep 15 21:09:45.808575 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.808496    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69"} err="failed to get container status \"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69\": rpc error: code = NotFound desc = could not find container \"f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69\": container with ID starting with f9922a59e1bbb28975e7bc6804be7130a21ce7ea8662698d5ec33cf7a300db69 not found: ID does not exist"
Sep 15 21:09:45.808575 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.808516    2579 scope.go:117] "RemoveContainer" containerID="36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"
Sep 15 21:09:45.815203 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls\") pod \"46a81999-7bf8-43ff-b62c-8d43046805e2\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") "
Sep 15 21:09:45.815297 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815243    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca\") pod \"46a81999-7bf8-43ff-b62c-8d43046805e2\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") "
Sep 15 21:09:45.815360 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fgp6\" (UniqueName: \"kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6\") pod \"74366d23-b564-4e72-b80e-694bd75ab64b\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") "
Sep 15 21:09:45.815360 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815340    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs\") pod \"46a81999-7bf8-43ff-b62c-8d43046805e2\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") "
Sep 15 21:09:45.815470 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume\") pod \"74366d23-b564-4e72-b80e-694bd75ab64b\" (UID: \"74366d23-b564-4e72-b80e-694bd75ab64b\") "
Sep 15 21:09:45.815523 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxmwb\" (UniqueName: \"kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb\") pod \"46a81999-7bf8-43ff-b62c-8d43046805e2\" (UID: \"46a81999-7bf8-43ff-b62c-8d43046805e2\") "
Sep 15 21:09:45.815718 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815652    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "46a81999-7bf8-43ff-b62c-8d43046805e2" (UID: "46a81999-7bf8-43ff-b62c-8d43046805e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:09:45.815816 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815755    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74366d23-b564-4e72-b80e-694bd75ab64b" (UID: "74366d23-b564-4e72-b80e-694bd75ab64b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:09:45.815907 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815884    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hts9d\" (UniqueName: \"kubernetes.io/projected/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-kube-api-access-hts9d\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815914    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46a81999-7bf8-43ff-b62c-8d43046805e2-openshift-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815929    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815944    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815956    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815969    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74366d23-b564-4e72-b80e-694bd75ab64b-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.815994 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815983    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.816502 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.815996    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.817303 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.817264    2579 scope.go:117] "RemoveContainer" containerID="36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"
Sep 15 21:09:45.817622 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:09:45.817597    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b\": container with ID starting with 36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b not found: ID does not exist" containerID="36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"
Sep 15 21:09:45.817718 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.817630    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b"} err="failed to get container status \"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b\": rpc error: code = NotFound desc = could not find container \"36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b\": container with ID starting with 36b2f0e6870b36c55407abe64d51c36a36fe52b1422451f2699a4e1e85c8083b not found: ID does not exist"
Sep 15 21:09:45.817718 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.817646    2579 scope.go:117] "RemoveContainer" containerID="bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"
Sep 15 21:09:45.817840 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.817800    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6" (OuterVolumeSpecName: "kube-api-access-6fgp6") pod "74366d23-b564-4e72-b80e-694bd75ab64b" (UID: "74366d23-b564-4e72-b80e-694bd75ab64b"). InnerVolumeSpecName "kube-api-access-6fgp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:09:45.818187 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.818153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "46a81999-7bf8-43ff-b62c-8d43046805e2" (UID: "46a81999-7bf8-43ff-b62c-8d43046805e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:09:45.818392 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.818371    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "46a81999-7bf8-43ff-b62c-8d43046805e2" (UID: "46a81999-7bf8-43ff-b62c-8d43046805e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:09:45.818474 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.818388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb" (OuterVolumeSpecName: "kube-api-access-cxmwb") pod "46a81999-7bf8-43ff-b62c-8d43046805e2" (UID: "46a81999-7bf8-43ff-b62c-8d43046805e2"). InnerVolumeSpecName "kube-api-access-cxmwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:09:45.822133 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.822110    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:09:45.825752 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.825730    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-728877/auth-test-model-728877-kserve-5b4957d6dc-w6pvq"]
Sep 15 21:09:45.826725 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.826712    2579 scope.go:117] "RemoveContainer" containerID="bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"
Sep 15 21:09:45.826978 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:09:45.826962    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6\": container with ID starting with bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6 not found: ID does not exist" containerID="bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"
Sep 15 21:09:45.827076 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.826984    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6"} err="failed to get container status \"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6\": rpc error: code = NotFound desc = could not find container \"bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6\": container with ID starting with bd8b9913ddd7af5ca8846be65caef62eb113cb15b8e0dd21932d25135d489ab6 not found: ID does not exist"
Sep 15 21:09:45.917253 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.917219    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-maas-api-tls\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.917253 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.917246    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgp6\" (UniqueName: \"kubernetes.io/projected/74366d23-b564-4e72-b80e-694bd75ab64b-kube-api-access-6fgp6\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.917253 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.917257    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46a81999-7bf8-43ff-b62c-8d43046805e2-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:45.917488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:45.917269    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxmwb\" (UniqueName: \"kubernetes.io/projected/46a81999-7bf8-43ff-b62c-8d43046805e2-kube-api-access-cxmwb\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:09:46.121802 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:46.121775    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 21:09:46.128155 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:46.128127    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-16bde4-6764c54ccb-fmszj"]
Sep 15 21:09:46.140307 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:46.140282    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 21:09:46.143245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:46.143221    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-16bde4-548bd8549d-hspc6"]
Sep 15 21:09:47.674024 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:47.673978    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46a81999-7bf8-43ff-b62c-8d43046805e2" path="/var/lib/kubelet/pods/46a81999-7bf8-43ff-b62c-8d43046805e2/volumes"
Sep 15 21:09:47.674397 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:47.674346    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74366d23-b564-4e72-b80e-694bd75ab64b" path="/var/lib/kubelet/pods/74366d23-b564-4e72-b80e-694bd75ab64b/volumes"
Sep 15 21:09:47.674685 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:47.674672    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" path="/var/lib/kubelet/pods/d5ee3f14-f61b-40fc-ae7a-67582e8c7b69/volumes"
Sep 15 21:09:52.667618 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:52.667574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:52.668101 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:52.667647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:52.680827 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:52.680807    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:09:52.848040 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:09:52.847998    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:10:00.365689 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:00.365650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:10:19.671605 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:19.671569    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:10:33.465988 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:33.465947    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:10:48.971771 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:48.971727    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:10:50.075962 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:50.075927    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:10:59.565573 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:10:59.565534    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:11:05.096903 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.096832    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" podUID="3f580886-725f-4254-a248-c6505abbf8c6" containerName="main" containerID="cri-o://b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410" gracePeriod=45
Sep 15 21:11:05.347939 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.347874    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:11:05.448650 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpn52\" (UniqueName: \"kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.448650 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.448897 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.448897 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.448897 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.448897 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448818    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs\") pod \"3f580886-725f-4254-a248-c6505abbf8c6\" (UID: \"3f580886-725f-4254-a248-c6505abbf8c6\") "
Sep 15 21:11:05.449150 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.448966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home" (OuterVolumeSpecName: "home") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:05.449150 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.449064    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:05.449150 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.449113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:05.449150 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.449126    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:05.451127 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.451104    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52" (OuterVolumeSpecName: "kube-api-access-lpn52") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "kube-api-access-lpn52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:11:05.451127 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.451107    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm" (OuterVolumeSpecName: "dshm") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:11:05.451274 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.451127    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3f580886-725f-4254-a248-c6505abbf8c6" (UID: "3f580886-725f-4254-a248-c6505abbf8c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:11:05.550098 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.550061    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f580886-725f-4254-a248-c6505abbf8c6-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:05.550098 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.550092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn52\" (UniqueName: \"kubernetes.io/projected/3f580886-725f-4254-a248-c6505abbf8c6-kube-api-access-lpn52\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:05.550315 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.550121    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:05.550315 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.550132    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:05.550315 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:05.550140    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f580886-725f-4254-a248-c6505abbf8c6-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:11:06.187304 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.187268    2579 generic.go:356] "Generic (PLEG): container finished" podID="3f580886-725f-4254-a248-c6505abbf8c6" containerID="b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410" exitCode=0
Sep 15 21:11:06.187798 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.187320    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"
Sep 15 21:11:06.187798 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.187319    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" event={"ID":"3f580886-725f-4254-a248-c6505abbf8c6","Type":"ContainerDied","Data":"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"}
Sep 15 21:11:06.187798 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.187452    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w" event={"ID":"3f580886-725f-4254-a248-c6505abbf8c6","Type":"ContainerDied","Data":"ff17441aaa919fb99da821ccca5539310dab7400870bd091da71928063f6edeb"}
Sep 15 21:11:06.187798 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.187481    2579 scope.go:117] "RemoveContainer" containerID="b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"
Sep 15 21:11:06.198326 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.197842    2579 scope.go:117] "RemoveContainer" containerID="b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"
Sep 15 21:11:06.198451 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:11:06.198374    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410\": container with ID starting with b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410 not found: ID does not exist" containerID="b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"
Sep 15 21:11:06.198451 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.198405    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410"} err="failed to get container status \"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410\": rpc error: code = NotFound desc = could not find container \"b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410\": container with ID starting with b115d84d00bc0aeb0aad5adf053cba2cc79da1fbe4cbae4d98730ea1fccef410 not found: ID does not exist"
Sep 15 21:11:06.205142 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.205118    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:11:06.215571 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:06.215543    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-280fb4/sub-test-model-280fb4-kserve-9cdb898dc-8xn5w"]
Sep 15 21:11:07.668204 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:07.668172    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f580886-725f-4254-a248-c6505abbf8c6" path="/var/lib/kubelet/pods/3f580886-725f-4254-a248-c6505abbf8c6/volumes"
Sep 15 21:11:14.664957 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:14.664877    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:11:29.978355 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:29.978317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:11:44.370860 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:44.370823    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:11:58.575505 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:11:58.575471    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:12:10.878402 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878367    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878712    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46a81999-7bf8-43ff-b62c-8d43046805e2" containerName="maas-api"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878722    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="46a81999-7bf8-43ff-b62c-8d43046805e2" containerName="maas-api"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878743    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerName="main"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878750    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerName="main"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878759    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f580886-725f-4254-a248-c6505abbf8c6" containerName="main"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878764    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f580886-725f-4254-a248-c6505abbf8c6" containerName="main"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74366d23-b564-4e72-b80e-694bd75ab64b" containerName="payload-pre-processing"
Sep 15 21:12:10.878818 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878779    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="74366d23-b564-4e72-b80e-694bd75ab64b" containerName="payload-pre-processing"
Sep 15 21:12:10.879128 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878844    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="46a81999-7bf8-43ff-b62c-8d43046805e2" containerName="maas-api"
Sep 15 21:12:10.879128 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878852    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ee3f14-f61b-40fc-ae7a-67582e8c7b69" containerName="main"
Sep 15 21:12:10.879128 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878858    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f580886-725f-4254-a248-c6505abbf8c6" containerName="main"
Sep 15 21:12:10.879128 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.878864    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="74366d23-b564-4e72-b80e-694bd75ab64b" containerName="payload-pre-processing"
Sep 15 21:12:10.882306 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.882286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:10.885048 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.885031    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-219de8\""
Sep 15 21:12:10.893734 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.893712    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:12:10.952962 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.952929    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:12:10.956246 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.956232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:10.958815 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.958795    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-219de8-dockercfg-484dv\""
Sep 15 21:12:10.966512 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:10.966483    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:12:11.034625 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.034587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.034625 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.034629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvwr\" (UniqueName: \"kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.136314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.136209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvwr\" (UniqueName: \"kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.136314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.136264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.136576 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.136356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm9t8\" (UniqueName: \"kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.136647 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.136592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.137402 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.137383    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.144615 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.144595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvwr\" (UniqueName: \"kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr\") pod \"payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.192722 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.192684    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:11.237964 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.237928    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.238175 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.237977    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm9t8\" (UniqueName: \"kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.238668 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.238644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.247109 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.246425    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm9t8\" (UniqueName: \"kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8\") pod \"payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.267442 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.267410    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:11.464415 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.464394    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:12:11.466422 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:12:11.466388    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a399ffb_1056_45f6_aa12_509aff8f573c.slice/crio-0a903b48d5e14d3ad48a5a53caac068dd1166818121447a14383154418a0bd52 WatchSource:0}: Error finding container 0a903b48d5e14d3ad48a5a53caac068dd1166818121447a14383154418a0bd52: Status 404 returned error can't find the container with id 0a903b48d5e14d3ad48a5a53caac068dd1166818121447a14383154418a0bd52
Sep 15 21:12:11.503579 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.503542    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" event={"ID":"7a399ffb-1056-45f6-aa12-509aff8f573c","Type":"ContainerStarted","Data":"0a903b48d5e14d3ad48a5a53caac068dd1166818121447a14383154418a0bd52"}
Sep 15 21:12:11.546091 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:11.545849    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:12:11.549340 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:12:11.549313    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81bb61_4696_47dc_b344_bc835b49321e.slice/crio-9611ef3431a43ad00e992749d56d35b7593063a4381e61389a1a22b4c2d75bc6 WatchSource:0}: Error finding container 9611ef3431a43ad00e992749d56d35b7593063a4381e61389a1a22b4c2d75bc6: Status 404 returned error can't find the container with id 9611ef3431a43ad00e992749d56d35b7593063a4381e61389a1a22b4c2d75bc6
Sep 15 21:12:12.509739 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.509697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" event={"ID":"7a399ffb-1056-45f6-aa12-509aff8f573c","Type":"ContainerStarted","Data":"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"}
Sep 15 21:12:12.510256 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.509766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:12.511324 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.511300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" event={"ID":"9b81bb61-4696-47dc-b344-bc835b49321e","Type":"ContainerStarted","Data":"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"}
Sep 15 21:12:12.511397 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.511344    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" event={"ID":"9b81bb61-4696-47dc-b344-bc835b49321e","Type":"ContainerStarted","Data":"9611ef3431a43ad00e992749d56d35b7593063a4381e61389a1a22b4c2d75bc6"}
Sep 15 21:12:12.511397 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.511377    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:12.526138 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.526080    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" podStartSLOduration=2.526066155 podStartE2EDuration="2.526066155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:12.524824324 +0000 UTC m=+2123.311948355" watchObservedRunningTime="2026-09-15 21:12:12.526066155 +0000 UTC m=+2123.313190181"
Sep 15 21:12:12.541018 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:12.540970    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" podStartSLOduration=2.540956615 podStartE2EDuration="2.540956615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:12:12.539414845 +0000 UTC m=+2123.326538912" watchObservedRunningTime="2026-09-15 21:12:12.540956615 +0000 UTC m=+2123.328080674"
Sep 15 21:12:14.072218 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:14.072183    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:12:23.518220 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:23.518184    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:12:23.520626 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:23.518345    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:12:45.179584 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:12:45.179492    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:13:13.050257 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:13.050229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:13:16.961023 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.960976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:13:16.965064 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.965041    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:16.968771 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.968750    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-219de8\"/\"default-dockercfg-p8n7h\""
Sep 15 21:13:16.968771 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.968766    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-219de8\"/\"kube-root-ca.crt\""
Sep 15 21:13:16.968971 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.968827    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-219de8\"/\"openshift-service-ca.crt\""
Sep 15 21:13:16.968971 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.968770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-219de8\"/\"ipp-route-219de8-kserve-self-signed-certs\""
Sep 15 21:13:16.975298 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:16.975276    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:13:17.045020 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.044985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.045189 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.045065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.045189 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.045085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.045189 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.045115    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.045300 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.045202    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.045300 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.045240    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmnk\" (UniqueName: \"kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145699 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145666    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmnk\" (UniqueName: \"kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.145878 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.145818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.146243 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.146209    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.146360 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.146253    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.146360 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.146314    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.148204 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.148186    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.148569 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.148549    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.158915 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.158892    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmnk\" (UniqueName: \"kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk\") pod \"ipp-route-219de8-kserve-95b859867-d6kvc\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") " pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.276291 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.276193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:17.551202 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.551175    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:13:17.553506 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:13:17.553476    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d880317_6b31_4239_9cdd_81152eb02dc0.slice/crio-47f2d5d3b66b29c65301ab267b06610d0e7565d8bd19d4ab7d3bbd59cb65a531 WatchSource:0}: Error finding container 47f2d5d3b66b29c65301ab267b06610d0e7565d8bd19d4ab7d3bbd59cb65a531: Status 404 returned error can't find the container with id 47f2d5d3b66b29c65301ab267b06610d0e7565d8bd19d4ab7d3bbd59cb65a531
Sep 15 21:13:17.843322 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.843229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" event={"ID":"8d880317-6b31-4239-9cdd-81152eb02dc0","Type":"ContainerStarted","Data":"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"}
Sep 15 21:13:17.843322 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.843271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" event={"ID":"8d880317-6b31-4239-9cdd-81152eb02dc0","Type":"ContainerStarted","Data":"47f2d5d3b66b29c65301ab267b06610d0e7565d8bd19d4ab7d3bbd59cb65a531"}
Sep 15 21:13:17.860489 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:17.860435    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" podStartSLOduration=1.8604151519999998 podStartE2EDuration="1.860415152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:17.859061873 +0000 UTC m=+2188.646185902" watchObservedRunningTime="2026-09-15 21:13:17.860415152 +0000 UTC m=+2188.647539183"
Sep 15 21:13:20.876105 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:20.876074    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:13:27.276569 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:27.276535    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:27.277080 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:27.276589    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:27.289266 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:27.289240    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:27.901486 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:27.901457    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:13:37.472410 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:37.472375    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:13:43.851889 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:43.851857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt_7a399ffb-1056-45f6-aa12-509aff8f573c/payload-processing/0.log"
Sep 15 21:13:43.969704 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:43.969669    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:13:51.030909 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.030876    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:13:51.034680 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.034662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.037448 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.037430    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-a6f49f\""
Sep 15 21:13:51.037525 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.037513    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-a6f49f-dockercfg-t87h4\""
Sep 15 21:13:51.050132 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.050099    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:13:51.062848 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.062817    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:13:51.066388 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.066357    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.066491 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.066400    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.066491 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.066454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msfld\" (UniqueName: \"kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.078374 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.078350    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:13:51.167570 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.167530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.167805 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.167601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.167805 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.167727    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msfld\" (UniqueName: \"kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.167805 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.167770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.168452 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.168426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.178666 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.178632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msfld\" (UniqueName: \"kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld\") pod \"payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.268423 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.268380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.268619 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.268516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.269204 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.269181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.280426 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.280398    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8\") pod \"payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.351314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.351217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:51.378093 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.378058    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:51.631726 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.631694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:13:51.634758 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:13:51.634732    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e4028d_dfcb_448c_93ff_85fcc006a466.slice/crio-97983d910b58369b2a6f001e7bd472f420e7e18583fd77257b3f4d822c14e0be WatchSource:0}: Error finding container 97983d910b58369b2a6f001e7bd472f420e7e18583fd77257b3f4d822c14e0be: Status 404 returned error can't find the container with id 97983d910b58369b2a6f001e7bd472f420e7e18583fd77257b3f4d822c14e0be
Sep 15 21:13:51.669724 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:51.669696    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:13:51.671075 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:13:51.671046    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5364d9ca_feab_4edf_a0cd_bd54ffc6410f.slice/crio-e5dea25741b8bfb3bc05f7c8c821deddb38c38ec53bfb28c30a6d82a6032b35e WatchSource:0}: Error finding container e5dea25741b8bfb3bc05f7c8c821deddb38c38ec53bfb28c30a6d82a6032b35e: Status 404 returned error can't find the container with id e5dea25741b8bfb3bc05f7c8c821deddb38c38ec53bfb28c30a6d82a6032b35e
Sep 15 21:13:52.014866 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.014756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" event={"ID":"56e4028d-dfcb-448c-93ff-85fcc006a466","Type":"ContainerStarted","Data":"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"}
Sep 15 21:13:52.015056 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.014930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:13:52.015056 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.014977    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" event={"ID":"56e4028d-dfcb-448c-93ff-85fcc006a466","Type":"ContainerStarted","Data":"97983d910b58369b2a6f001e7bd472f420e7e18583fd77257b3f4d822c14e0be"}
Sep 15 21:13:52.016467 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.016441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" event={"ID":"5364d9ca-feab-4edf-a0cd-bd54ffc6410f","Type":"ContainerStarted","Data":"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"}
Sep 15 21:13:52.016608 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.016494    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:13:52.016608 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.016517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" event={"ID":"5364d9ca-feab-4edf-a0cd-bd54ffc6410f","Type":"ContainerStarted","Data":"e5dea25741b8bfb3bc05f7c8c821deddb38c38ec53bfb28c30a6d82a6032b35e"}
Sep 15 21:13:52.030926 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.030877    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" podStartSLOduration=1.030859977 podStartE2EDuration="1.030859977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:52.02961766 +0000 UTC m=+2222.816741690" watchObservedRunningTime="2026-09-15 21:13:52.030859977 +0000 UTC m=+2222.817984128"
Sep 15 21:13:52.047022 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:52.046977    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" podStartSLOduration=1.046963331 podStartE2EDuration="1.046963331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:13:52.045905812 +0000 UTC m=+2222.833029843" watchObservedRunningTime="2026-09-15 21:13:52.046963331 +0000 UTC m=+2222.834087360"
Sep 15 21:13:56.863785 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:13:56.863749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:14:03.024459 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:03.024430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:14:03.024846 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:03.024497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:14:19.401394 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.401358    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:14:19.401913 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.401666    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" podUID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" containerName="payload-processing" containerID="cri-o://7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126" gracePeriod=30
Sep 15 21:14:19.409734 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.409708    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:14:19.409974 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.409946    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" podUID="56e4028d-dfcb-448c-93ff-85fcc006a466" containerName="payload-pre-processing" containerID="cri-o://0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f" gracePeriod=30
Sep 15 21:14:19.683405 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.683351    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:14:19.708481 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.708450    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:14:19.751968 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.751930    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume\") pod \"56e4028d-dfcb-448c-93ff-85fcc006a466\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") "
Sep 15 21:14:19.752422 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.752388    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "56e4028d-dfcb-448c-93ff-85fcc006a466" (UID: "56e4028d-dfcb-448c-93ff-85fcc006a466"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:19.852655 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.852618    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8\") pod \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") "
Sep 15 21:14:19.852857 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.852723    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msfld\" (UniqueName: \"kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld\") pod \"56e4028d-dfcb-448c-93ff-85fcc006a466\" (UID: \"56e4028d-dfcb-448c-93ff-85fcc006a466\") "
Sep 15 21:14:19.852857 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.852747    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume\") pod \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\" (UID: \"5364d9ca-feab-4edf-a0cd-bd54ffc6410f\") "
Sep 15 21:14:19.852987 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.852961    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56e4028d-dfcb-448c-93ff-85fcc006a466-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:19.853268 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.853239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5364d9ca-feab-4edf-a0cd-bd54ffc6410f" (UID: "5364d9ca-feab-4edf-a0cd-bd54ffc6410f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:19.854957 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.854927    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld" (OuterVolumeSpecName: "kube-api-access-msfld") pod "56e4028d-dfcb-448c-93ff-85fcc006a466" (UID: "56e4028d-dfcb-448c-93ff-85fcc006a466"). InnerVolumeSpecName "kube-api-access-msfld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:19.855077 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.854975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8" (OuterVolumeSpecName: "kube-api-access-7mbt8") pod "5364d9ca-feab-4edf-a0cd-bd54ffc6410f" (UID: "5364d9ca-feab-4edf-a0cd-bd54ffc6410f"). InnerVolumeSpecName "kube-api-access-7mbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:19.953740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.953652    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msfld\" (UniqueName: \"kubernetes.io/projected/56e4028d-dfcb-448c-93ff-85fcc006a466-kube-api-access-msfld\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:19.953740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.953680    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:19.953740 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:19.953689    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/5364d9ca-feab-4edf-a0cd-bd54ffc6410f-kube-api-access-7mbt8\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:20.157633 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.157602    2579 generic.go:356] "Generic (PLEG): container finished" podID="56e4028d-dfcb-448c-93ff-85fcc006a466" containerID="0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f" exitCode=0
Sep 15 21:14:20.157835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.157648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"
Sep 15 21:14:20.157835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.157683    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" event={"ID":"56e4028d-dfcb-448c-93ff-85fcc006a466","Type":"ContainerDied","Data":"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"}
Sep 15 21:14:20.157835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.157733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk" event={"ID":"56e4028d-dfcb-448c-93ff-85fcc006a466","Type":"ContainerDied","Data":"97983d910b58369b2a6f001e7bd472f420e7e18583fd77257b3f4d822c14e0be"}
Sep 15 21:14:20.157835 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.157755    2579 scope.go:117] "RemoveContainer" containerID="0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"
Sep 15 21:14:20.159190 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.159168    2579 generic.go:356] "Generic (PLEG): container finished" podID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" containerID="7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126" exitCode=0
Sep 15 21:14:20.159279 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.159212    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"
Sep 15 21:14:20.159279 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.159238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" event={"ID":"5364d9ca-feab-4edf-a0cd-bd54ffc6410f","Type":"ContainerDied","Data":"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"}
Sep 15 21:14:20.159382 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.159286    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk" event={"ID":"5364d9ca-feab-4edf-a0cd-bd54ffc6410f","Type":"ContainerDied","Data":"e5dea25741b8bfb3bc05f7c8c821deddb38c38ec53bfb28c30a6d82a6032b35e"}
Sep 15 21:14:20.167661 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.167642    2579 scope.go:117] "RemoveContainer" containerID="0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"
Sep 15 21:14:20.167960 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:14:20.167943    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f\": container with ID starting with 0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f not found: ID does not exist" containerID="0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"
Sep 15 21:14:20.168044 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.167970    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f"} err="failed to get container status \"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f\": rpc error: code = NotFound desc = could not find container \"0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f\": container with ID starting with 0d3bed2fe9b208216d9d0c50953810dc79eee02101fa8a58187de41e5243074f not found: ID does not exist"
Sep 15 21:14:20.168044 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.167985    2579 scope.go:117] "RemoveContainer" containerID="7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"
Sep 15 21:14:20.177262 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.177234    2579 scope.go:117] "RemoveContainer" containerID="7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"
Sep 15 21:14:20.177521 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:14:20.177493    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126\": container with ID starting with 7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126 not found: ID does not exist" containerID="7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"
Sep 15 21:14:20.177634 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.177530    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126"} err="failed to get container status \"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126\": rpc error: code = NotFound desc = could not find container \"7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126\": container with ID starting with 7f12322b9ad380b9b60ea43ee9c5ef24b1a2121f084706f79068b343493fa126 not found: ID does not exist"
Sep 15 21:14:20.186734 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.186704    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:14:20.193145 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.193112    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-a6f49f-dc4465fdc-7kxbk"]
Sep 15 21:14:20.205647 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.205569    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:14:20.209576 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:20.209551    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-a6f49f-6f8598dc94-vkmkk"]
Sep 15 21:14:21.669121 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:21.669085    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" path="/var/lib/kubelet/pods/5364d9ca-feab-4edf-a0cd-bd54ffc6410f/volumes"
Sep 15 21:14:21.669566 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:21.669518    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e4028d-dfcb-448c-93ff-85fcc006a466" path="/var/lib/kubelet/pods/56e4028d-dfcb-448c-93ff-85fcc006a466/volumes"
Sep 15 21:14:23.861688 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:23.861651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:14:42.768651 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:42.768622    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:14:45.269470 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.269438    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:14:45.269933 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.269767    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" podUID="7a399ffb-1056-45f6-aa12-509aff8f573c" containerName="payload-processing" containerID="cri-o://964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f" gracePeriod=30
Sep 15 21:14:45.274998 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.274968    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:14:45.275248 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.275220    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" podUID="9b81bb61-4696-47dc-b344-bc835b49321e" containerName="payload-pre-processing" containerID="cri-o://90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016" gracePeriod=30
Sep 15 21:14:45.545279 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.545200    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:14:45.582035 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.581976    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:14:45.691737 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.691709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume\") pod \"9b81bb61-4696-47dc-b344-bc835b49321e\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") "
Sep 15 21:14:45.691913 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.691744    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fhvwr\" (UniqueName: \"kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr\") pod \"7a399ffb-1056-45f6-aa12-509aff8f573c\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") "
Sep 15 21:14:45.691913 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.691796    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm9t8\" (UniqueName: \"kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8\") pod \"9b81bb61-4696-47dc-b344-bc835b49321e\" (UID: \"9b81bb61-4696-47dc-b344-bc835b49321e\") "
Sep 15 21:14:45.691913 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.691847    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume\") pod \"7a399ffb-1056-45f6-aa12-509aff8f573c\" (UID: \"7a399ffb-1056-45f6-aa12-509aff8f573c\") "
Sep 15 21:14:45.692229 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.692126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b81bb61-4696-47dc-b344-bc835b49321e" (UID: "9b81bb61-4696-47dc-b344-bc835b49321e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:45.692294 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.692270    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7a399ffb-1056-45f6-aa12-509aff8f573c" (UID: "7a399ffb-1056-45f6-aa12-509aff8f573c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:14:45.694174 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.694151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr" (OuterVolumeSpecName: "kube-api-access-fhvwr") pod "7a399ffb-1056-45f6-aa12-509aff8f573c" (UID: "7a399ffb-1056-45f6-aa12-509aff8f573c"). InnerVolumeSpecName "kube-api-access-fhvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:45.694287 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.694200    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8" (OuterVolumeSpecName: "kube-api-access-nm9t8") pod "9b81bb61-4696-47dc-b344-bc835b49321e" (UID: "9b81bb61-4696-47dc-b344-bc835b49321e"). InnerVolumeSpecName "kube-api-access-nm9t8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:14:45.793573 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.793539    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm9t8\" (UniqueName: \"kubernetes.io/projected/9b81bb61-4696-47dc-b344-bc835b49321e-kube-api-access-nm9t8\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:45.793573 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.793572    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a399ffb-1056-45f6-aa12-509aff8f573c-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:45.793837 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.793585    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b81bb61-4696-47dc-b344-bc835b49321e-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:45.793837 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:45.793600    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvwr\" (UniqueName: \"kubernetes.io/projected/7a399ffb-1056-45f6-aa12-509aff8f573c-kube-api-access-fhvwr\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:14:46.295140 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.295112    2579 generic.go:356] "Generic (PLEG): container finished" podID="9b81bb61-4696-47dc-b344-bc835b49321e" containerID="90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016" exitCode=0
Sep 15 21:14:46.295607 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.295170    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"
Sep 15 21:14:46.295607 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.295197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" event={"ID":"9b81bb61-4696-47dc-b344-bc835b49321e","Type":"ContainerDied","Data":"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"}
Sep 15 21:14:46.295607 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.295248    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt" event={"ID":"9b81bb61-4696-47dc-b344-bc835b49321e","Type":"ContainerDied","Data":"9611ef3431a43ad00e992749d56d35b7593063a4381e61389a1a22b4c2d75bc6"}
Sep 15 21:14:46.295607 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.295281    2579 scope.go:117] "RemoveContainer" containerID="90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"
Sep 15 21:14:46.296762 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.296674    2579 generic.go:356] "Generic (PLEG): container finished" podID="7a399ffb-1056-45f6-aa12-509aff8f573c" containerID="964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f" exitCode=0
Sep 15 21:14:46.296762 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.296722    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"
Sep 15 21:14:46.296762 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.296733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" event={"ID":"7a399ffb-1056-45f6-aa12-509aff8f573c","Type":"ContainerDied","Data":"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"}
Sep 15 21:14:46.297057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.296812    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt" event={"ID":"7a399ffb-1056-45f6-aa12-509aff8f573c","Type":"ContainerDied","Data":"0a903b48d5e14d3ad48a5a53caac068dd1166818121447a14383154418a0bd52"}
Sep 15 21:14:46.305000 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.304980    2579 scope.go:117] "RemoveContainer" containerID="90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"
Sep 15 21:14:46.305305 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:14:46.305285    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016\": container with ID starting with 90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016 not found: ID does not exist" containerID="90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"
Sep 15 21:14:46.305371 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.305312    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016"} err="failed to get container status \"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016\": rpc error: code = NotFound desc = could not find container \"90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016\": container with ID starting with 90cda3b8e5f9fc037712986aa7c1bfede2030542e7be2086554483997d084016 not found: ID does not exist"
Sep 15 21:14:46.305371 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.305330    2579 scope.go:117] "RemoveContainer" containerID="964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"
Sep 15 21:14:46.313599 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.313577    2579 scope.go:117] "RemoveContainer" containerID="964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"
Sep 15 21:14:46.313871 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:14:46.313851    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f\": container with ID starting with 964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f not found: ID does not exist" containerID="964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"
Sep 15 21:14:46.313930 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.313877    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f"} err="failed to get container status \"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f\": rpc error: code = NotFound desc = could not find container \"964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f\": container with ID starting with 964aea3afc48b802e6a3c03907c4c41c9e9dc0d233f7fe801bb7c4419030b22f not found: ID does not exist"
Sep 15 21:14:46.320281 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.320257    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:14:46.323552 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.323530    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-219de8-7cc4c4f7d7-snlkt"]
Sep 15 21:14:46.333918 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.333892    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:14:46.337174 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:46.337153    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-219de8-7499dbb454-lnjtt"]
Sep 15 21:14:47.668266 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:47.668230    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a399ffb-1056-45f6-aa12-509aff8f573c" path="/var/lib/kubelet/pods/7a399ffb-1056-45f6-aa12-509aff8f573c/volumes"
Sep 15 21:14:47.668746 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:47.668706    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b81bb61-4696-47dc-b344-bc835b49321e" path="/var/lib/kubelet/pods/9b81bb61-4696-47dc-b344-bc835b49321e/volumes"
Sep 15 21:14:50.812339 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:50.812306    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:14:56.168158 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:14:56.168118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:15:00.136314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136277    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"]
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136619    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a399ffb-1056-45f6-aa12-509aff8f573c" containerName="payload-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136630    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a399ffb-1056-45f6-aa12-509aff8f573c" containerName="payload-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136641    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e4028d-dfcb-448c-93ff-85fcc006a466" containerName="payload-pre-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136647    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e4028d-dfcb-448c-93ff-85fcc006a466" containerName="payload-pre-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136657    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" containerName="payload-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136663    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" containerName="payload-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136682    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81bb61-4696-47dc-b344-bc835b49321e" containerName="payload-pre-processing"
Sep 15 21:15:00.136702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136687    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81bb61-4696-47dc-b344-bc835b49321e" containerName="payload-pre-processing"
Sep 15 21:15:00.136952 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136740    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a399ffb-1056-45f6-aa12-509aff8f573c" containerName="payload-processing"
Sep 15 21:15:00.136952 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136748    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81bb61-4696-47dc-b344-bc835b49321e" containerName="payload-pre-processing"
Sep 15 21:15:00.136952 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136754    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5364d9ca-feab-4edf-a0cd-bd54ffc6410f" containerName="payload-processing"
Sep 15 21:15:00.136952 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.136760    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e4028d-dfcb-448c-93ff-85fcc006a466" containerName="payload-pre-processing"
Sep 15 21:15:00.145355 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.145297    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"]
Sep 15 21:15:00.145795 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.145767    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"
Sep 15 21:15:00.148800 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.148770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-t4pz9\""
Sep 15 21:15:00.151129 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.150681    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"]
Sep 15 21:15:00.151129 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.150768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"
Sep 15 21:15:00.152177 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.152158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"]
Sep 15 21:15:00.161578 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.161558    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"
Sep 15 21:15:00.168934 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.168903    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"
Sep 15 21:15:00.440644 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.440624    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"]
Sep 15 21:15:00.442712 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:15:00.442686    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bdd880_0f6d_4589_a812_ff8c1caa8f56.slice/crio-e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72 WatchSource:0}: Error finding container e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72: Status 404 returned error can't find the container with id e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72
Sep 15 21:15:00.451987 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:00.451967    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"]
Sep 15 21:15:00.455318 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:15:00.455295    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b97d1e6_ee07_44dc_be71_a5b438d0a668.slice/crio-c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84 WatchSource:0}: Error finding container c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84: Status 404 returned error can't find the container with id c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84
Sep 15 21:15:01.373335 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.373296    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r" event={"ID":"3b97d1e6-ee07-44dc-be71-a5b438d0a668","Type":"ContainerStarted","Data":"c63d1791637456093656abc49e1e9ce61b71a76535681bd5a0e555f86885a669"}
Sep 15 21:15:01.373702 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.373357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r" event={"ID":"3b97d1e6-ee07-44dc-be71-a5b438d0a668","Type":"ContainerStarted","Data":"c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84"}
Sep 15 21:15:01.374861 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.374839    2579 generic.go:356] "Generic (PLEG): container finished" podID="b5bdd880-0f6d-4589-a812-ff8c1caa8f56" containerID="e64c853404fed6fdd485a3f514906a2bf450b4acc1726254d5d2f3eb5c5e0846" exitCode=0
Sep 15 21:15:01.374967 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.374917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt" event={"ID":"b5bdd880-0f6d-4589-a812-ff8c1caa8f56","Type":"ContainerDied","Data":"e64c853404fed6fdd485a3f514906a2bf450b4acc1726254d5d2f3eb5c5e0846"}
Sep 15 21:15:01.374967 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.374963    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt" event={"ID":"b5bdd880-0f6d-4589-a812-ff8c1caa8f56","Type":"ContainerStarted","Data":"e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72"}
Sep 15 21:15:01.390294 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:01.390242    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r" podStartSLOduration=1.390226275 podStartE2EDuration="1.390226275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:15:01.389495771 +0000 UTC m=+2292.176619792" watchObservedRunningTime="2026-09-15 21:15:01.390226275 +0000 UTC m=+2292.177350350"
Sep 15 21:15:02.380959 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:02.380925    2579 generic.go:356] "Generic (PLEG): container finished" podID="3b97d1e6-ee07-44dc-be71-a5b438d0a668" containerID="c63d1791637456093656abc49e1e9ce61b71a76535681bd5a0e555f86885a669" exitCode=0
Sep 15 21:15:02.381388 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:02.381027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r" event={"ID":"3b97d1e6-ee07-44dc-be71-a5b438d0a668","Type":"ContainerDied","Data":"c63d1791637456093656abc49e1e9ce61b71a76535681bd5a0e555f86885a669"}
Sep 15 21:15:02.517425 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:02.517402    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"
Sep 15 21:15:03.387072 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:03.387040    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"
Sep 15 21:15:03.387488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:03.387077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt" event={"ID":"b5bdd880-0f6d-4589-a812-ff8c1caa8f56","Type":"ContainerDied","Data":"e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72"}
Sep 15 21:15:03.387488 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:03.387161    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e30e12a08900a1c34b09e43b59f75696addea0e177126eb2c4fee0f7e39f6e72"
Sep 15 21:15:03.522237 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:03.522214    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"
Sep 15 21:15:04.392996 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:04.392964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r" event={"ID":"3b97d1e6-ee07-44dc-be71-a5b438d0a668","Type":"ContainerDied","Data":"c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84"}
Sep 15 21:15:04.392996 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:04.392975    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"
Sep 15 21:15:04.393462 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:04.393035    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9eb0bc3beba36c54c5093321c23ab5ff0b3482265a87de384b3673f88a96b84"
Sep 15 21:15:05.832412 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:05.832375    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" podUID="8d880317-6b31-4239-9cdd-81152eb02dc0" containerName="main" containerID="cri-o://5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954" gracePeriod=45
Sep 15 21:15:06.096320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.096250    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:15:06.189731 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlmnk\" (UniqueName: \"kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.189924 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189741    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.189924 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189781    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.189924 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189807    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.189924 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189827    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.190167 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.189997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir\") pod \"8d880317-6b31-4239-9cdd-81152eb02dc0\" (UID: \"8d880317-6b31-4239-9cdd-81152eb02dc0\") "
Sep 15 21:15:06.190167 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home" (OuterVolumeSpecName: "home") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:06.190167 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190134    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:06.190320 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:06.190416 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190398    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-home\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.190475 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190420    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-tmp-dir\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.190475 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.190435    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-model-cache\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.192198 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.192170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:15:06.192198 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.192187    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm" (OuterVolumeSpecName: "dshm") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 21:15:06.192369 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.192240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk" (OuterVolumeSpecName: "kube-api-access-dlmnk") pod "8d880317-6b31-4239-9cdd-81152eb02dc0" (UID: "8d880317-6b31-4239-9cdd-81152eb02dc0"). InnerVolumeSpecName "kube-api-access-dlmnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:15:06.291708 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.291655    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmnk\" (UniqueName: \"kubernetes.io/projected/8d880317-6b31-4239-9cdd-81152eb02dc0-kube-api-access-dlmnk\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.291708 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.291700    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d880317-6b31-4239-9cdd-81152eb02dc0-dshm\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.291708 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.291712    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d880317-6b31-4239-9cdd-81152eb02dc0-tls-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:15:06.404108 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.404080    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d880317-6b31-4239-9cdd-81152eb02dc0" containerID="5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954" exitCode=0
Sep 15 21:15:06.404268 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.404125    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"
Sep 15 21:15:06.404268 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.404125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" event={"ID":"8d880317-6b31-4239-9cdd-81152eb02dc0","Type":"ContainerDied","Data":"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"}
Sep 15 21:15:06.404268 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.404170    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc" event={"ID":"8d880317-6b31-4239-9cdd-81152eb02dc0","Type":"ContainerDied","Data":"47f2d5d3b66b29c65301ab267b06610d0e7565d8bd19d4ab7d3bbd59cb65a531"}
Sep 15 21:15:06.404268 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.404196    2579 scope.go:117] "RemoveContainer" containerID="5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"
Sep 15 21:15:06.413939 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.413923    2579 scope.go:117] "RemoveContainer" containerID="5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"
Sep 15 21:15:06.414203 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:15:06.414183    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954\": container with ID starting with 5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954 not found: ID does not exist" containerID="5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"
Sep 15 21:15:06.414267 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.414211    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954"} err="failed to get container status \"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954\": rpc error: code = NotFound desc = could not find container \"5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954\": container with ID starting with 5725a1ac7cd35a26839aac30c429d4610eb712e28f75aa2414e7cdd0ac1b7954 not found: ID does not exist"
Sep 15 21:15:06.426504 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.426481    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:15:06.432024 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:06.431987    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-219de8/ipp-route-219de8-kserve-95b859867-d6kvc"]
Sep 15 21:15:07.668774 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:07.668738    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d880317-6b31-4239-9cdd-81152eb02dc0" path="/var/lib/kubelet/pods/8d880317-6b31-4239-9cdd-81152eb02dc0/volumes"
Sep 15 21:15:44.900812 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:44.900731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"]
Sep 15 21:15:44.907417 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:44.907386    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-7f09a7-29825115-km8qt"]
Sep 15 21:15:45.667952 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:45.667916    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5bdd880-0f6d-4589-a812-ff8c1caa8f56" path="/var/lib/kubelet/pods/b5bdd880-0f6d-4589-a812-ff8c1caa8f56/volumes"
Sep 15 21:15:48.166788 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:15:48.166755    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:16:05.965860 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:05.965824    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:16:21.204514 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.204474    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"]
Sep 15 21:16:21.208253 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.208228    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-728877-29825115-f6k7r"]
Sep 15 21:16:21.228953 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.228922    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:16:21.229210 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.229185    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" podUID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" containerName="payload-processing" containerID="cri-o://5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d" gracePeriod=30
Sep 15 21:16:21.239206 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.239183    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:16:21.239372 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.239354    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" podUID="39a2e6c1-73c3-4553-8692-139b29664c0f" containerName="payload-pre-processing" containerID="cri-o://4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320" gracePeriod=30
Sep 15 21:16:21.497461 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.497437    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:16:21.532832 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.532754    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:16:21.568352 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.568322    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume\") pod \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") "
Sep 15 21:16:21.568352 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.568359    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dbgp\" (UniqueName: \"kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp\") pod \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\" (UID: \"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df\") "
Sep 15 21:16:21.568799 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.568772    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" (UID: "b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:16:21.570999 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.570966    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp" (OuterVolumeSpecName: "kube-api-access-7dbgp") pod "b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" (UID: "b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df"). InnerVolumeSpecName "kube-api-access-7dbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:21.669337 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.669305    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b97d1e6-ee07-44dc-be71-a5b438d0a668" path="/var/lib/kubelet/pods/3b97d1e6-ee07-44dc-be71-a5b438d0a668/volumes"
Sep 15 21:16:21.669670 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.669653    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume\") pod \"39a2e6c1-73c3-4553-8692-139b29664c0f\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") "
Sep 15 21:16:21.669714 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.669700    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4klb\" (UniqueName: \"kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb\") pod \"39a2e6c1-73c3-4553-8692-139b29664c0f\" (UID: \"39a2e6c1-73c3-4553-8692-139b29664c0f\") "
Sep 15 21:16:21.669842 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.669829    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:21.669909 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.669842    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dbgp\" (UniqueName: \"kubernetes.io/projected/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df-kube-api-access-7dbgp\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:21.670053 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.670002    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "39a2e6c1-73c3-4553-8692-139b29664c0f" (UID: "39a2e6c1-73c3-4553-8692-139b29664c0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:16:21.671889 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.671868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb" (OuterVolumeSpecName: "kube-api-access-w4klb") pod "39a2e6c1-73c3-4553-8692-139b29664c0f" (UID: "39a2e6c1-73c3-4553-8692-139b29664c0f"). InnerVolumeSpecName "kube-api-access-w4klb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:21.771184 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.771092    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4klb\" (UniqueName: \"kubernetes.io/projected/39a2e6c1-73c3-4553-8692-139b29664c0f-kube-api-access-w4klb\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:21.771184 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.771121    2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39a2e6c1-73c3-4553-8692-139b29664c0f-plugins-config-volume\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:21.785479 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.785447    2579 generic.go:356] "Generic (PLEG): container finished" podID="39a2e6c1-73c3-4553-8692-139b29664c0f" containerID="4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320" exitCode=0
Sep 15 21:16:21.785626 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.785494    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"
Sep 15 21:16:21.785626 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.785523    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" event={"ID":"39a2e6c1-73c3-4553-8692-139b29664c0f","Type":"ContainerDied","Data":"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"}
Sep 15 21:16:21.785626 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.785587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx" event={"ID":"39a2e6c1-73c3-4553-8692-139b29664c0f","Type":"ContainerDied","Data":"f6592c8b3ab34f3427fe993f70b43077e345b78195eb0ef7bf3f46704bdbf5f8"}
Sep 15 21:16:21.785626 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.785613    2579 scope.go:117] "RemoveContainer" containerID="4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"
Sep 15 21:16:21.786923 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.786894    2579 generic.go:356] "Generic (PLEG): container finished" podID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" containerID="5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d" exitCode=0
Sep 15 21:16:21.787069 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.786936    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"
Sep 15 21:16:21.787069 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.786975    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" event={"ID":"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df","Type":"ContainerDied","Data":"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"}
Sep 15 21:16:21.787196 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.787070    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs" event={"ID":"b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df","Type":"ContainerDied","Data":"937c7017b1b89c021c1d1311f7ffe618a7fd3349142c5e4adb7083a263cface3"}
Sep 15 21:16:21.796112 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.796092    2579 scope.go:117] "RemoveContainer" containerID="4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"
Sep 15 21:16:21.796433 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:16:21.796415    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320\": container with ID starting with 4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320 not found: ID does not exist" containerID="4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"
Sep 15 21:16:21.796514 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.796444    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320"} err="failed to get container status \"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320\": rpc error: code = NotFound desc = could not find container \"4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320\": container with ID starting with 4a951236579fd25ac70c2fa10db5586fc870675b25e156ab539d25ba4e077320 not found: ID does not exist"
Sep 15 21:16:21.796514 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.796461    2579 scope.go:117] "RemoveContainer" containerID="5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"
Sep 15 21:16:21.805259 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.805223    2579 scope.go:117] "RemoveContainer" containerID="5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"
Sep 15 21:16:21.805593 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:16:21.805571    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d\": container with ID starting with 5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d not found: ID does not exist" containerID="5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"
Sep 15 21:16:21.805700 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.805608    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d"} err="failed to get container status \"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d\": rpc error: code = NotFound desc = could not find container \"5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d\": container with ID starting with 5b0ed49558cc89ff509ca074a07141e2cc4935e6d42cceddb4e3d9fb5c05680d not found: ID does not exist"
Sep 15 21:16:21.807386 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.807368    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:16:21.811666 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.811645    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-728877-85f676d575-cb4qs"]
Sep 15 21:16:21.824288 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.824254    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:16:21.829477 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:21.829450    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-728877-5b64f59b5d-98mzx"]
Sep 15 21:16:23.669583 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:23.669546    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a2e6c1-73c3-4553-8692-139b29664c0f" path="/var/lib/kubelet/pods/39a2e6c1-73c3-4553-8692-139b29664c0f/volumes"
Sep 15 21:16:23.669963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:23.669888    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" path="/var/lib/kubelet/pods/b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df/volumes"
Sep 15 21:16:24.075283 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:24.075186    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:16:32.056324 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.056287    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 21:16:32.056991 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.056500    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" podUID="eed9c28d-ab12-4def-86c4-0e29b531acc1" containerName="manager" containerID="cri-o://9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc" gracePeriod=30
Sep 15 21:16:32.311184 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.311110    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 21:16:32.379779 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.379746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs\") pod \"eed9c28d-ab12-4def-86c4-0e29b531acc1\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") "
Sep 15 21:16:32.379978 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.379786    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert\") pod \"eed9c28d-ab12-4def-86c4-0e29b531acc1\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") "
Sep 15 21:16:32.379978 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.379842    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwbqr\" (UniqueName: \"kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr\") pod \"eed9c28d-ab12-4def-86c4-0e29b531acc1\" (UID: \"eed9c28d-ab12-4def-86c4-0e29b531acc1\") "
Sep 15 21:16:32.382078 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.381995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "eed9c28d-ab12-4def-86c4-0e29b531acc1" (UID: "eed9c28d-ab12-4def-86c4-0e29b531acc1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:16:32.382196 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.382075    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr" (OuterVolumeSpecName: "kube-api-access-fwbqr") pod "eed9c28d-ab12-4def-86c4-0e29b531acc1" (UID: "eed9c28d-ab12-4def-86c4-0e29b531acc1"). InnerVolumeSpecName "kube-api-access-fwbqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:16:32.382271 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.382256    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "eed9c28d-ab12-4def-86c4-0e29b531acc1" (UID: "eed9c28d-ab12-4def-86c4-0e29b531acc1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:16:32.480791 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.480752    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwbqr\" (UniqueName: \"kubernetes.io/projected/eed9c28d-ab12-4def-86c4-0e29b531acc1-kube-api-access-fwbqr\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:32.480791 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.480786    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:32.480791 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.480795    2579 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eed9c28d-ab12-4def-86c4-0e29b531acc1-webhook-cert\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:16:32.845113 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.845077    2579 generic.go:356] "Generic (PLEG): container finished" podID="eed9c28d-ab12-4def-86c4-0e29b531acc1" containerID="9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc" exitCode=0
Sep 15 21:16:32.845290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.845128    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4"
Sep 15 21:16:32.845290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.845134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" event={"ID":"eed9c28d-ab12-4def-86c4-0e29b531acc1","Type":"ContainerDied","Data":"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"}
Sep 15 21:16:32.845290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.845177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-smxg4" event={"ID":"eed9c28d-ab12-4def-86c4-0e29b531acc1","Type":"ContainerDied","Data":"0f484d4855b1a7e28a5db6ac696f97ff32facbe39459bbdc48a10b0ca6d25ec6"}
Sep 15 21:16:32.845290 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.845198    2579 scope.go:117] "RemoveContainer" containerID="9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"
Sep 15 21:16:32.854834 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.854816    2579 scope.go:117] "RemoveContainer" containerID="9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"
Sep 15 21:16:32.855154 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:16:32.855133    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc\": container with ID starting with 9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc not found: ID does not exist" containerID="9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"
Sep 15 21:16:32.855221 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.855165    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc"} err="failed to get container status \"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc\": rpc error: code = NotFound desc = could not find container \"9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc\": container with ID starting with 9cc7453c67be259b07e2a899413e7ea48ffe51aa22f80bee6ec92d76e21f4fbc not found: ID does not exist"
Sep 15 21:16:32.874323 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.874296    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 21:16:32.887851 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:32.887824    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-smxg4"]
Sep 15 21:16:33.477978 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:33.477944    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:16:33.668194 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:16:33.668160    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eed9c28d-ab12-4def-86c4-0e29b531acc1" path="/var/lib/kubelet/pods/eed9c28d-ab12-4def-86c4-0e29b531acc1/volumes"
Sep 15 21:17:55.168847 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:55.168810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:17:56.741804 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.741771    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-t942c"]
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742132    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5bdd880-0f6d-4589-a812-ff8c1caa8f56" containerName="cleanup"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742145    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5bdd880-0f6d-4589-a812-ff8c1caa8f56" containerName="cleanup"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742159    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d880317-6b31-4239-9cdd-81152eb02dc0" containerName="main"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742164    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d880317-6b31-4239-9cdd-81152eb02dc0" containerName="main"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742170    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" containerName="payload-processing"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742175    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" containerName="payload-processing"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742187    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eed9c28d-ab12-4def-86c4-0e29b531acc1" containerName="manager"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742192    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eed9c28d-ab12-4def-86c4-0e29b531acc1" containerName="manager"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742200    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b97d1e6-ee07-44dc-be71-a5b438d0a668" containerName="cleanup"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742204    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b97d1e6-ee07-44dc-be71-a5b438d0a668" containerName="cleanup"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742214    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a2e6c1-73c3-4553-8692-139b29664c0f" containerName="payload-pre-processing"
Sep 15 21:17:56.742245 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742219    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a2e6c1-73c3-4553-8692-139b29664c0f" containerName="payload-pre-processing"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742273    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a2e6c1-73c3-4553-8692-139b29664c0f" containerName="payload-pre-processing"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742280    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5bdd880-0f6d-4589-a812-ff8c1caa8f56" containerName="cleanup"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742288    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eed9c28d-ab12-4def-86c4-0e29b531acc1" containerName="manager"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742293    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b43cf0e1-7b6e-4f1a-ac89-e40a6f5d08df" containerName="payload-processing"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742299    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d880317-6b31-4239-9cdd-81152eb02dc0" containerName="main"
Sep 15 21:17:56.742694 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.742304    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b97d1e6-ee07-44dc-be71-a5b438d0a668" containerName="cleanup"
Sep 15 21:17:56.745220 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.745200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.749225 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.749202    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 15 21:17:56.749370 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.749203    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 15 21:17:56.749370 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.749252    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 15 21:17:56.749370 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.749292    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-shjx4\""
Sep 15 21:17:56.756281 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.756245    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-t942c"]
Sep 15 21:17:56.812505 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.812470    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcw2q\" (UniqueName: \"kubernetes.io/projected/70ffe3c3-c304-4b27-bdad-0c151468ee88-kube-api-access-rcw2q\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.812705 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.812559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-metrics-certs\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.812705 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.812627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-webhook-cert\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.913658 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.913623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-metrics-certs\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.913658 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.913663    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-webhook-cert\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.913867 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.913726    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcw2q\" (UniqueName: \"kubernetes.io/projected/70ffe3c3-c304-4b27-bdad-0c151468ee88-kube-api-access-rcw2q\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.916339 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.916310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-webhook-cert\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.916452 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.916361    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70ffe3c3-c304-4b27-bdad-0c151468ee88-metrics-certs\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:56.922388 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:56.922364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcw2q\" (UniqueName: \"kubernetes.io/projected/70ffe3c3-c304-4b27-bdad-0c151468ee88-kube-api-access-rcw2q\") pod \"maas-controller-68d9b8fcfd-t942c\" (UID: \"70ffe3c3-c304-4b27-bdad-0c151468ee88\") " pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:57.055296 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:57.055201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:57.348032 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:57.347990    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68d9b8fcfd-t942c"]
Sep 15 21:17:57.350412 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:17:57.350378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ffe3c3_c304_4b27_bdad_0c151468ee88.slice/crio-66bae514c7a4772c6bfcb6578fbe876da5eb97b4fefb5d28ccf002b8b172fdc5 WatchSource:0}: Error finding container 66bae514c7a4772c6bfcb6578fbe876da5eb97b4fefb5d28ccf002b8b172fdc5: Status 404 returned error can't find the container with id 66bae514c7a4772c6bfcb6578fbe876da5eb97b4fefb5d28ccf002b8b172fdc5
Sep 15 21:17:57.351784 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:57.351766    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 21:17:58.243762 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:58.243722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-t942c" event={"ID":"70ffe3c3-c304-4b27-bdad-0c151468ee88","Type":"ContainerStarted","Data":"fad29a38a3598bc3fdb7899cc9ba6d8d6ff898e294fe324a89bb81556abb3ced"}
Sep 15 21:17:58.244295 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:58.243773    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68d9b8fcfd-t942c" event={"ID":"70ffe3c3-c304-4b27-bdad-0c151468ee88","Type":"ContainerStarted","Data":"66bae514c7a4772c6bfcb6578fbe876da5eb97b4fefb5d28ccf002b8b172fdc5"}
Sep 15 21:17:58.244295 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:58.243867    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:17:58.263122 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:17:58.263077    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-68d9b8fcfd-t942c" podStartSLOduration=1.976916352 podStartE2EDuration="2.263060147s" totalImagesPullingTime="286.143795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:17:58.262502588 +0000 UTC m=+2469.049626618" watchObservedRunningTime="2026-09-15 21:17:58.263060147 +0000 UTC m=+2469.050184176"
Sep 15 21:18:09.251281 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:09.251244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68d9b8fcfd-t942c"
Sep 15 21:18:18.980208 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:18.980165    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:18:27.467148 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:27.467113    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:18:35.272145 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:35.272107    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:18:50.381324 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:50.381290    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:18:58.381031 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:18:58.378906    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:20:24.771230 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:20:24.771193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:20:32.970708 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:20:32.970671    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:20:41.266652 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:20:41.266616    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:20:45.875732 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:20:45.875695    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:09.270260 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:09.270218    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:32.773664 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:32.773632    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:40.475514 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:40.475477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:47.670252 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:47.670171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:50.025030 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:50.024988    2579 scope.go:117] "RemoveContainer" containerID="e64c853404fed6fdd485a3f514906a2bf450b4acc1726254d5d2f3eb5c5e0846"
Sep 15 21:21:50.035212 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:50.035190    2579 scope.go:117] "RemoveContainer" containerID="c63d1791637456093656abc49e1e9ce61b71a76535681bd5a0e555f86885a669"
Sep 15 21:21:55.971872 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:55.971826    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:21:56.334981 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.334898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"]
Sep 15 21:21:56.338406 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.338384    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.346540 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.346513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"]
Sep 15 21:21:56.446635 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.446603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h844\" (UniqueName: \"kubernetes.io/projected/028e4599-445e-4ca0-bf18-d65d53097685-kube-api-access-7h844\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.446815 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.446660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-metrics-certs\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.446815 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.446768    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4599-445e-4ca0-bf18-d65d53097685-openshift-service-ca\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.446900 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.446812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-maas-api-tls\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.547640 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.547605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h844\" (UniqueName: \"kubernetes.io/projected/028e4599-445e-4ca0-bf18-d65d53097685-kube-api-access-7h844\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.547825 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.547668    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-metrics-certs\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.547825 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.547722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4599-445e-4ca0-bf18-d65d53097685-openshift-service-ca\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.547825 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.547746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-maas-api-tls\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.548507 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.548432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/028e4599-445e-4ca0-bf18-d65d53097685-openshift-service-ca\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.550260 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.550233    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-maas-api-tls\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.550260 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.550258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/028e4599-445e-4ca0-bf18-d65d53097685-metrics-certs\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.555847 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.555823    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h844\" (UniqueName: \"kubernetes.io/projected/028e4599-445e-4ca0-bf18-d65d53097685-kube-api-access-7h844\") pod \"maas-api-569cf8d545-qj22f\" (UID: \"028e4599-445e-4ca0-bf18-d65d53097685\") " pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.651176 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.651141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:56.924685 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:56.924660    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"]
Sep 15 21:21:56.926651 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:21:56.926626    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028e4599_445e_4ca0_bf18_d65d53097685.slice/crio-2625ef515b6863bf418028de61437171ac2cf55437f6ae56dad064c79bf66e3a WatchSource:0}: Error finding container 2625ef515b6863bf418028de61437171ac2cf55437f6ae56dad064c79bf66e3a: Status 404 returned error can't find the container with id 2625ef515b6863bf418028de61437171ac2cf55437f6ae56dad064c79bf66e3a
Sep 15 21:21:57.356839 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:57.356798    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f" event={"ID":"028e4599-445e-4ca0-bf18-d65d53097685","Type":"ContainerStarted","Data":"a952f196576925ad4aceaa280926525d47e56163f28a40204ab48b64420e7079"}
Sep 15 21:21:57.357299 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:57.356880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f" event={"ID":"028e4599-445e-4ca0-bf18-d65d53097685","Type":"ContainerStarted","Data":"2625ef515b6863bf418028de61437171ac2cf55437f6ae56dad064c79bf66e3a"}
Sep 15 21:21:57.357299 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:57.356940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:21:57.374075 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:21:57.373993    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f" podStartSLOduration=1.025271514 podStartE2EDuration="1.373973016s" totalImagesPullingTime="348.701502ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 21:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:21:57.373381037 +0000 UTC m=+2708.160505064" watchObservedRunningTime="2026-09-15 21:21:57.373973016 +0000 UTC m=+2708.161097046"
Sep 15 21:22:03.367128 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:03.367097    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-569cf8d545-qj22f"
Sep 15 21:22:03.415588 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:03.415554    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 21:22:03.415803 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:03.415784    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerName="maas-api" containerID="cri-o://cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd" gracePeriod=30
Sep 15 21:22:03.669148 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:03.669110    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:22:03.777579 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:03.777527    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.32:8443/health\": dial tcp 10.133.0.32:8443: connect: connection refused"
Sep 15 21:22:04.663411 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.663385    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 21:22:04.721518 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.721428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca\") pod \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") "
Sep 15 21:22:04.721518 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.721461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls\") pod \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") "
Sep 15 21:22:04.721518 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.721490    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsrg8\" (UniqueName: \"kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8\") pod \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") "
Sep 15 21:22:04.721803 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.721548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs\") pod \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\" (UID: \"5fa90bf3-e629-4b15-90a3-5799acb35a6e\") "
Sep 15 21:22:04.721872 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.721835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5fa90bf3-e629-4b15-90a3-5799acb35a6e" (UID: "5fa90bf3-e629-4b15-90a3-5799acb35a6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 21:22:04.723809 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.723780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5fa90bf3-e629-4b15-90a3-5799acb35a6e" (UID: "5fa90bf3-e629-4b15-90a3-5799acb35a6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:22:04.723936 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.723804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5fa90bf3-e629-4b15-90a3-5799acb35a6e" (UID: "5fa90bf3-e629-4b15-90a3-5799acb35a6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 21:22:04.723936 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.723847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8" (OuterVolumeSpecName: "kube-api-access-hsrg8") pod "5fa90bf3-e629-4b15-90a3-5799acb35a6e" (UID: "5fa90bf3-e629-4b15-90a3-5799acb35a6e"). InnerVolumeSpecName "kube-api-access-hsrg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 21:22:04.822248 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.822214    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsrg8\" (UniqueName: \"kubernetes.io/projected/5fa90bf3-e629-4b15-90a3-5799acb35a6e-kube-api-access-hsrg8\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:22:04.822248 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.822241    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-metrics-certs\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:22:04.822248 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.822251    2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa90bf3-e629-4b15-90a3-5799acb35a6e-openshift-service-ca\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:22:04.822480 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:04.822260    2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa90bf3-e629-4b15-90a3-5799acb35a6e-maas-api-tls\") on node \"ip-10-0-138-122.ec2.internal\" DevicePath \"\""
Sep 15 21:22:05.395875 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.395836    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerID="cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd" exitCode=0
Sep 15 21:22:05.396086 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.395884    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh"
Sep 15 21:22:05.396086 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.395902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" event={"ID":"5fa90bf3-e629-4b15-90a3-5799acb35a6e","Type":"ContainerDied","Data":"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"}
Sep 15 21:22:05.396086 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.395952    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746c646f77-884fh" event={"ID":"5fa90bf3-e629-4b15-90a3-5799acb35a6e","Type":"ContainerDied","Data":"7a93eec1a2cfb7cf948551ab839d8005bffaf4a1d53c60cb6380ac0427571a66"}
Sep 15 21:22:05.396086 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.395982    2579 scope.go:117] "RemoveContainer" containerID="cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"
Sep 15 21:22:05.405711 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.405694    2579 scope.go:117] "RemoveContainer" containerID="cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"
Sep 15 21:22:05.405986 ip-10-0-138-122 kubenswrapper[2579]: E0915 21:22:05.405966    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd\": container with ID starting with cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd not found: ID does not exist" containerID="cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"
Sep 15 21:22:05.406072 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.405995    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd"} err="failed to get container status \"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd\": rpc error: code = NotFound desc = could not find container \"cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd\": container with ID starting with cff35b6e3f5f9dc237d6b0222e383eebc6ef882bbc7b40dc3b54f792c67012bd not found: ID does not exist"
Sep 15 21:22:05.416861 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.416839    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 21:22:05.422616 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.422592    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-746c646f77-884fh"]
Sep 15 21:22:05.668173 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:05.668095    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" path="/var/lib/kubelet/pods/5fa90bf3-e629-4b15-90a3-5799acb35a6e/volumes"
Sep 15 21:22:11.966305 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:11.966268    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:22:18.277691 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:18.277650    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-klcvg"]
Sep 15 21:22:27.458764 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:27.458729    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w8bf5_b38566c6-7cef-40ee-9960-d5aef815ab3b/manager/0.log"
Sep 15 21:22:27.709171 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:27.709093    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d9b8fcfd-t942c_70ffe3c3-c304-4b27-bdad-0c151468ee88/manager/0.log"
Sep 15 21:22:28.225281 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:28.225252    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-9wbv6_046286e0-b330-4533-a2e2-829caf6cf5be/copy-manifests/0.log"
Sep 15 21:22:28.232173 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:28.232141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-9wbv6_046286e0-b330-4533-a2e2-829caf6cf5be/manager/0.log"
Sep 15 21:22:28.848143 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:28.848119    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-569cf8d545-qj22f_028e4599-445e-4ca0-bf18-d65d53097685/maas-api/0.log"
Sep 15 21:22:30.168265 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:30.168237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-d6w9v_5289a7c6-c9a0-4230-bb82-84707da7bb41/manager/0.log"
Sep 15 21:22:30.287127 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:30.287099    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bc9ht_3c4236ea-8bca-486c-877d-fe01d9f2d9be/manager/0.log"
Sep 15 21:22:30.662220 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:30.662196    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-klcvg_0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e/limitador/0.log"
Sep 15 21:22:31.308550 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:31.308520    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9gzfd_0dc80540-cdb8-4e1e-bab3-2b5c5f493332/discovery/0.log"
Sep 15 21:22:31.423959 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:31.423931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-2xvl7_7fb4dafb-3c2b-4509-91b7-dd823ac52ee4/kube-auth-proxy/0.log"
Sep 15 21:22:31.917933 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:31.917902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:22:32.510859 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:32.510826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k4868_f20fac15-e5b9-4988-b162-2328087e0946/main/0.log"
Sep 15 21:22:32.752839 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:32.752813    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k4969_7de42155-f36a-40ef-92f8-6849d66fc638/main/0.log"
Sep 15 21:22:32.873544 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:32.873517    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6dk8t_8626f72c-a7d6-4861-850e-ff02ad41aa51/main/0.log"
Sep 15 21:22:41.897518 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:41.897484    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lrldg_a5c75bc2-7d31-4663-a3cd-31b435dcecd4/global-pull-secret-syncer/0.log"
Sep 15 21:22:42.017648 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:42.017614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rpv5g_a4a6f3b5-90b7-44a5-865b-d27c6dc6254d/konnectivity-agent/0.log"
Sep 15 21:22:42.059636 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:42.059603    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-122.ec2.internal_81ca5a9bd4d1a43083fabcf1bcff5aa1/haproxy/0.log"
Sep 15 21:22:46.866630 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:46.866599    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-d6w9v_5289a7c6-c9a0-4230-bb82-84707da7bb41/manager/0.log"
Sep 15 21:22:46.890698 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:46.890677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-bc9ht_3c4236ea-8bca-486c-877d-fe01d9f2d9be/manager/0.log"
Sep 15 21:22:46.975564 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:46.975539    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-klcvg_0115e4b9-ad5e-44a1-8184-3dc7dd9cbb4e/limitador/0.log"
Sep 15 21:22:49.084023 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:49.083931    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fdcq_a649fb8f-5155-4a38-9979-098122c8990b/node-exporter/0.log"
Sep 15 21:22:49.104926 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:49.104897    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fdcq_a649fb8f-5155-4a38-9979-098122c8990b/kube-rbac-proxy/0.log"
Sep 15 21:22:49.127378 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:49.127356    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5fdcq_a649fb8f-5155-4a38-9979-098122c8990b/init-textfile/0.log"
Sep 15 21:22:49.580639 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:49.580609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cdlks_0dccda17-f48c-46c1-bcba-ea7e6b6bd355/prometheus-operator/0.log"
Sep 15 21:22:49.599931 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:49.599902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cdlks_0dccda17-f48c-46c1-bcba-ea7e6b6bd355/kube-rbac-proxy/0.log"
Sep 15 21:22:51.760717 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.760681    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"]
Sep 15 21:22:51.761148 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.761080    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerName="maas-api"
Sep 15 21:22:51.761148 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.761093    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerName="maas-api"
Sep 15 21:22:51.761246 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.761173    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa90bf3-e629-4b15-90a3-5799acb35a6e" containerName="maas-api"
Sep 15 21:22:51.765466 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.765448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.768205 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.768181    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tcx6s\"/\"default-dockercfg-8qfvh\""
Sep 15 21:22:51.769203 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.769183    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"kube-root-ca.crt\""
Sep 15 21:22:51.769303 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.769190    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tcx6s\"/\"openshift-service-ca.crt\""
Sep 15 21:22:51.773605 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.773579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"]
Sep 15 21:22:51.854118 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.854080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-podres\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.854118 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.854121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-proc\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.854369 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.854194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-sys\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.854369 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.854219    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-lib-modules\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.854369 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.854252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcc4x\" (UniqueName: \"kubernetes.io/projected/3858fa7e-8694-4413-8bb1-3a07486218e0-kube-api-access-xcc4x\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955713 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-podres\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955713 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-proc\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955772    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-sys\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-lib-modules\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcc4x\" (UniqueName: \"kubernetes.io/projected/3858fa7e-8694-4413-8bb1-3a07486218e0-kube-api-access-xcc4x\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-proc\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955845    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-podres\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-sys\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.955963 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.955916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3858fa7e-8694-4413-8bb1-3a07486218e0-lib-modules\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:51.964674 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:51.964645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcc4x\" (UniqueName: \"kubernetes.io/projected/3858fa7e-8694-4413-8bb1-3a07486218e0-kube-api-access-xcc4x\") pod \"perf-node-gather-daemonset-b82qh\" (UID: \"3858fa7e-8694-4413-8bb1-3a07486218e0\") " pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:52.075843 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.075751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:52.327416 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.327348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6c75569b7c-lg46q_7a1f8f2e-e8e2-4f47-9fcf-35bfa7b85dea/console/0.log"
Sep 15 21:22:52.344596 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.344573    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"]
Sep 15 21:22:52.346485 ip-10-0-138-122 kubenswrapper[2579]: W0915 21:22:52.346459    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3858fa7e_8694_4413_8bb1_3a07486218e0.slice/crio-f28e0a4061bfc0ab081b742410015ccb85bfdf990fb1d0ba43f814e7f9643077 WatchSource:0}: Error finding container f28e0a4061bfc0ab081b742410015ccb85bfdf990fb1d0ba43f814e7f9643077: Status 404 returned error can't find the container with id f28e0a4061bfc0ab081b742410015ccb85bfdf990fb1d0ba43f814e7f9643077
Sep 15 21:22:52.362566 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.362549    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-klgxq_8de405a3-c83e-49ed-91bd-7267bdc52c11/download-server/0.log"
Sep 15 21:22:52.618037 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.617929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh" event={"ID":"3858fa7e-8694-4413-8bb1-3a07486218e0","Type":"ContainerStarted","Data":"165dbf14e054b347a7ae5d45e4bec47d2d9433fea98898a080f30e8ca8bb6399"}
Sep 15 21:22:52.618037 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.617980    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh" event={"ID":"3858fa7e-8694-4413-8bb1-3a07486218e0","Type":"ContainerStarted","Data":"f28e0a4061bfc0ab081b742410015ccb85bfdf990fb1d0ba43f814e7f9643077"}
Sep 15 21:22:52.618238 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.618053    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:22:52.634582 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:52.634532    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh" podStartSLOduration=1.634517299 podStartE2EDuration="1.634517299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 21:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 21:22:52.634203011 +0000 UTC m=+2763.421327050" watchObservedRunningTime="2026-09-15 21:22:52.634517299 +0000 UTC m=+2763.421641388"
Sep 15 21:22:54.127419 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:54.127394    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-26p5b_4e43d015-d108-49ce-9e1b-917ea3b35955/dns/0.log"
Sep 15 21:22:54.146915 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:54.146889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-26p5b_4e43d015-d108-49ce-9e1b-917ea3b35955/kube-rbac-proxy/0.log"
Sep 15 21:22:54.256269 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:54.256241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mzct6_df934d45-30b6-47d7-a704-ffbdd0b01465/dns-node-resolver/0.log"
Sep 15 21:22:54.766305 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:54.766262    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b69dc5c9c-2dkxx_2304952d-987b-4c91-923d-8d3a1f55fcbd/registry/0.log"
Sep 15 21:22:54.807519 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:54.807492    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dqclf_12f64684-4fa9-47db-b5c2-fbd466664932/node-ca/0.log"
Sep 15 21:22:56.174931 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:56.174898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9gzfd_0dc80540-cdb8-4e1e-bab3-2b5c5f493332/discovery/0.log"
Sep 15 21:22:56.197714 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:56.197690    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-2xvl7_7fb4dafb-3c2b-4509-91b7-dd823ac52ee4/kube-auth-proxy/0.log"
Sep 15 21:22:56.316883 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:56.316855    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:22:57.241918 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:57.241886    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-bpdfj_3c29b539-3db1-4827-9171-a832c12afb14/serve-healthcheck-canary/0.log"
Sep 15 21:22:57.913575 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:57.913544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x87rb_ae55e282-d1ba-4200-bb42-620685f1b1ee/kube-rbac-proxy/0.log"
Sep 15 21:22:57.931865 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:57.931837    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x87rb_ae55e282-d1ba-4200-bb42-620685f1b1ee/exporter/0.log"
Sep 15 21:22:57.951259 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:57.951216    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-x87rb_ae55e282-d1ba-4200-bb42-620685f1b1ee/extractor/0.log"
Sep 15 21:22:58.633477 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:22:58.633449    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tcx6s/perf-node-gather-daemonset-b82qh"
Sep 15 21:23:00.087524 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:00.087497    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-w8bf5_b38566c6-7cef-40ee-9960-d5aef815ab3b/manager/0.log"
Sep 15 21:23:00.503314 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:00.503281    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68d9b8fcfd-t942c_70ffe3c3-c304-4b27-bdad-0c151468ee88/manager/0.log"
Sep 15 21:23:00.787395 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:00.787370    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-9wbv6_046286e0-b330-4533-a2e2-829caf6cf5be/manager/0.log"
Sep 15 21:23:00.805931 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:00.805910    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-687b5f4fc4-9wbv6_046286e0-b330-4533-a2e2-829caf6cf5be/copy-manifests/0.log"
Sep 15 21:23:02.265864 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:02.265831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-24tww_0751e3b5-50da-4d2f-92ef-334587009b21/openshift-lws-operator/0.log"
Sep 15 21:23:02.383674 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:02.383642    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9gzfd_0dc80540-cdb8-4e1e-bab3-2b5c5f493332/discovery/0.log"
Sep 15 21:23:02.393076 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:02.393049    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-2xvl7_7fb4dafb-3c2b-4509-91b7-dd823ac52ee4/kube-auth-proxy/0.log"
Sep 15 21:23:02.473311 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:02.473279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:23:04.429130 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:04.429100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9gzfd_0dc80540-cdb8-4e1e-bab3-2b5c5f493332/discovery/0.log"
Sep 15 21:23:04.448568 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:04.448536    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-65544b76bd-2xvl7_7fb4dafb-3c2b-4509-91b7-dd823ac52ee4/kube-auth-proxy/0.log"
Sep 15 21:23:04.535057 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:04.535030    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-bn6qq_80b73a39-dc35-4151-b2f5-23bfbf56cadb/payload-processing/0.log"
Sep 15 21:23:09.074396 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.074368    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/kube-multus-additional-cni-plugins/0.log"
Sep 15 21:23:09.096561 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.096531    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/egress-router-binary-copy/0.log"
Sep 15 21:23:09.117199 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.117172    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/cni-plugins/0.log"
Sep 15 21:23:09.137785 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.137759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/bond-cni-plugin/0.log"
Sep 15 21:23:09.158917 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.158890    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/routeoverride-cni/0.log"
Sep 15 21:23:09.178179 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.178101    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/whereabouts-cni-bincopy/0.log"
Sep 15 21:23:09.197450 ip-10-0-138-122 kubenswrapper[2579]: I0915 21:23:09.197416    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-qwpwz_8dfdc110-4e9b-4aae-a0c7-9dd3029dfe05/whereabouts-cni/0.log"